Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 23.728648192s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:18:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 23.728698507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.178828548s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.178854456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.101692646s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.101901914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 14.094855072s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 14.094885309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.936906595s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.936940629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 18.472191342s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:14:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 18.472246867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.747632647s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:13:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.7476606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.905397227s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.905427624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.012018441s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:12:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.012046756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.780352389s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:12:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.780382036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.078639688s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:11:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.078687099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.006872643s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:10:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.006915214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.919319156s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:10:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.919343912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.772104402s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:09:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.772161943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.835598456s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:09:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.835630126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.906305876s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.906436705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.959283544s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.959315015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.014268986s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.014299355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.816004235s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:05:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.81603314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.795091497s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:05:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.795114259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.001215554s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:04:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.001247494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.645984128s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:04:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.646013355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.808888207s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.808915309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.927169791s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:02:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.927193766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.769295851s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:01:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.76932193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.017460318s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.01748781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.866283829s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.866347921s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.790128049s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 03:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.790161371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.134760133s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.13479028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.822725652s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:58:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.822758376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.390867319s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:57:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.390894019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.278580205s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:57:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 7.278609181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.817818289s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:57:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.817848396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.863234203s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.863264923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.92029995s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.920328926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.033079157s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:54:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.033134422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.54364474s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:54:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.543662445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.954758539s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.954785841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.034035166s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:52:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.034060815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.962517713s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.962562668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.851050966s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:50:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.851083306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.960089701s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.960116492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.013483878s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 6.013514635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.675882164s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 7.675914355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.745193385s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.745213564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.005335046s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 8.005355115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 10.791286815s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 10.791319327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.928477419s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:45:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.928506935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.985507593s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 5.985537578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.032330423s trace: https://jaeger.dev.storj.io/trace/6c1fe0e611b0604a FAIL (2025-12-17 02:44:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q) context deadline exceeded ======LOGS====== Dec 17 02:35:49.694751892 (275h6m49.565834937s) [upload=241197]: Starting upload Dec 17 02:35:49.694849047 (275h6m49.565932102s) [upload=241197]: Done starting upload Dec 17 02:35:49.694932817 (275h6m49.566015862s) [upload=241197]: Uploading segments... Dec 17 02:35:49.695198583 (275h6m49.566281628s) [upload=241197]: Got next segment. Inline: false Dec 17 02:35:49.695295427 (275h6m49.566378462s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 02:35:49.695303833 (275h6m49.566386868s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 02:35:49.942175689 (275h6m49.813258734s) [seg_pos={0 0}, upload=241197]: Begin upload segment... Dec 17 02:35:49.942230393 (275h6m49.813313428s) [seg_pos={0 0}, upload=241197]: Done begin upload segment. Dec 17 02:35:49.942317198 (275h6m49.813400243s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942328580 (275h6m49.813411625s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.942995349 (275h6m49.814078384s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943140667 (275h6m49.814223712s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943576737 (275h6m49.814659772s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943600973 (275h6m49.814684018s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.943979434 (275h6m49.815062469s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944411947 (275h6m49.815494992s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.944882523 (275h6m49.815965569s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945290660 (275h6m49.816373705s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.945751909 (275h6m49.816834944s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946223346 (275h6m49.817306381s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.946911156 (275h6m49.817994201s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.947697944 (275h6m49.818780989s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948436290 (275h6m49.819519335s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:49.948710984 (275h6m49.819794029s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Uploading piece... Dec 17 02:35:51.640521225 (275h6m51.51160426s) [upload=241197]: Next returned nil segment Dec 17 02:35:51.640564207 (275h6m51.511647262s) [upload=241197]: Waiting for error group managing segments... Dec 17 02:35:51.930827777 (275h6m51.801910822s) [noise=true , address=145.79.227.52:4445 , piece_id=H34BDCCNWDOC5UND2YFO4IQT2TSFWKJ5M7ZSXXX5CBM2VVXBARVA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933185758 (275h6m51.804268803s) [noise=true , address=145.79.227.39:4013 , piece_id=YHUOKQWYDTKHHFFBCTUQABIPG6HQBP2IDKXBDOMZL3J3W667D2BQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.933226715 (275h6m51.80430976s) [noise=true , address=145.79.227.48:4303 , piece_id=55XPFR3TBBFL2LZQBKY3UVLKM7M56H2PBT4FOQBAE6OIELDOEZHQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.936297684 (275h6m51.807380729s) [noise=true , address=145.79.227.34:4905 , piece_id=5LWUBKIPHE7J62OSSJM6J6QE5EVR7OUPT36GJTEH62SXLHNAIZQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946279801 (275h6m51.817362846s) [noise=true , address=145.79.227.47:4266 , piece_id=MYCO2FFVUT2QMHOMBOR7K25JOZLNH2DYWT6MRW3H2LF3QUQUC3BA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.946630498 (275h6m51.817713544s) [noise=true , address=145.79.227.33:4870 , piece_id=PALIZUEDLO7MEKFFYHMX6NDBCFQPZEPL5BPT5BSDSSKAXDQ7BS3A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.954181397 (275h6m51.825264432s) [noise=true , address=145.79.227.51:4409 , piece_id=2JANQNF5F5Z7AAF3NIJ2K5V6LZYTESTEX4ACBOL2ATLNXBLFF76Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:51.958311240 (275h6m51.829394285s) [noise=true , address=145.79.227.40:4050 , piece_id=47DO5LAQ5BGZYFFDWHWWYWKH6IDRBVPPFTTWCYCBPZHKXH6BV2WA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.094666835 (275h6m51.96574987s) [noise=true , address=77.90.31.18:4807 , piece_id=NMQAQBQWOHHTDMIP4UJ7CKDWRL6ETTCJCA7ROIBPFTMC6NE5I55Q, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.166532390 (275h6m52.037615435s) [noise=true , address=77.90.31.19:4842 , piece_id=NEUGN7PG3JTVJ7TEP5VFF4QV72HQOE54DUMSME3C6NBFYVUFXVVQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.190309254 (275h6m52.061392299s) [noise=true , address=45.91.116.10:4084 , piece_id=67QZU4WGKG7PTKG7OKJ7EBHLH2RI35YFVHQ4QC7LADE57LFWUINQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.191745568 (275h6m52.062828614s) [noise=true , address=45.91.116.12:4226 , piece_id=JMQQXYJDQI3VZKTJQUCWM7J2ULWEATLLRO4BQUJ34G4TLL5PPO5A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246797658 (275h6m52.117880703s) [noise=true , address=45.91.116.14:4190 , piece_id=TD3XXLZEEXM6OIU2LURWWDPOGI3VY7JOD2JANEX5ZQCZLCE3QIOA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: <nil> Dec 17 02:35:52.246830731 (275h6m52.117913776s) [seg_pos={0 0}, upload=241197]: Segment reached optimal threshold of 13 pieces. Dec 17 02:35:52.247301286 (275h6m52.118384321s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247308821 (275h6m52.118391856s) [noise=true , address=145.79.227.30:4761 , piece_id=ICEVVB7ZVJ4XXV6ME2TQUG2NDGC32K7ZX2M3HZQ5VBSRTP57FXQQ, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247499654 (275h6m52.118582689s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247506427 (275h6m52.118589472s) [noise=true , address=45.91.116.13:4153 , piece_id=2GCFARQ3IIKI3QXDONPWF6WLGYPJOIKTQ3UDZ6NNPAMBZZCOYK6A, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247828621 (275h6m52.118911666s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 02:35:52.247858938 (275h6m52.118941993s) [noise=true , address=45.91.116.7:4009 , piece_id=X7LO25NCZSVXFGBV36FU5EEAYLUWIXQNYNPLXO3Z3UNO73UTDCGA, seg_pos={0 0}, upload=241197]: Long tail cancellation due to optimal threshold Dec 17 02:35:52.247887683 (275h6m52.118970718s) [seg_pos={0 0}, upload=241197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 02:35:52.247980831 (275h6m52.119063876s) [upload=241197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 02:35:52.247988024 (275h6m52.119071069s) [upload=241197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 02:35:52.550785481 (275h6m52.421868526s) [upload=241197]: Done uploading segments... Dec 17 02:35:52.550806522 (275h6m52.421889567s) [upload=241197]: Upload finished. err: <nil> Duration: 7.032351282s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)