Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.934609459s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:06:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.934637403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.642545084s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:06:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.642572788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.312362807s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:05:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.31239069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.79473269s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:04:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.794749862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X............................................X..............] duration: 5.827240575s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.82726957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X............................................X.............] duration: 5.853135723s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:02:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.853189697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X............................................X............] duration: 5.940214878s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.940241479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.759218437s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 03:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.759254425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X............................................X...........] duration: 5.741656396s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.741683408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.52535954s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 03:00:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.525392704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.456392339s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 03:00:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.456432455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X............................................X..........] duration: 5.981016957s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 03:00:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.981050681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................................X.........] duration: 5.806651205s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.806681573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.601914936s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:59:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.60194819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............................................X........] duration: 5.723907343s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:59:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.723939846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.240401161s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:58:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.240436459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................................X.......] duration: 5.829082564s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:58:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.82911206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............................................X......] duration: 5.84092202s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:57:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.840947759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.540256667s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:57:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.540280171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.540042304s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:56:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.540097961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X............................................X.....] duration: 5.813306901s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:56:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.813339663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X............................................X....] duration: 5.775778245s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.775813253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.377046878s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:55:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.377075924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X............................................X...] duration: 5.769773918s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:55:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.769806449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.547657459s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:55:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.547691925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.648285172s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:54:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.648315751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X............................................X..] duration: 5.755042003s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.755072562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.64756918s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.647593356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X............................................X.] duration: 5.676978781s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:53:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.677012155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................................X] duration: 30.237118776s trace: https://jaeger.dev.storj.io/trace/5c121d70bd99253e FAIL (2025-12-26 02:52:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 26 02:52:19.731791723 (491h23m19.602874768s) [upload=431272]: Starting upload Dec 26 02:52:19.731848490 (491h23m19.602931526s) [upload=431272]: Done starting upload Dec 26 02:52:19.732033013 (491h23m19.603116048s) [upload=431272]: Uploading segments... Dec 26 02:52:19.732070955 (491h23m19.603154s) [upload=431272]: Got next segment. Inline: false Dec 26 02:52:19.732098547 (491h23m19.603181583s) [upload=431272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:52:19.732103427 (491h23m19.603186462s) [upload=431272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:52:19.973293532 (491h23m19.844376577s) [seg_pos={0 0}, upload=431272]: Begin upload segment... Dec 26 02:52:19.973352103 (491h23m19.844435138s) [seg_pos={0 0}, upload=431272]: Done begin upload segment. Dec 26 02:52:19.973407248 (491h23m19.844490283s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973416586 (491h23m19.844499631s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.973537346 (491h23m19.844620381s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974245295 (491h23m19.84532834s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.974722563 (491h23m19.845805608s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975206384 (491h23m19.846289419s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.975615052 (491h23m19.846698098s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976024011 (491h23m19.847107057s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.976520376 (491h23m19.847603421s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.977012163 (491h23m19.848095208s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978385388 (491h23m19.849468433s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.978614574 (491h23m19.849697619s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980036201 (491h23m19.851119246s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.980141902 (491h23m19.851224937s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.985930806 (491h23m19.857013871s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:19.991508236 (491h23m19.862591271s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Uploading piece... Dec 26 02:52:21.689082031 (491h23m21.560165076s) [upload=431272]: Next returned nil segment Dec 26 02:52:21.689107981 (491h23m21.560191026s) [upload=431272]: Waiting for error group managing segments... Dec 26 02:52:21.972478137 (491h23m21.843561172s) [noise=true , address=145.79.227.48:4301 , piece_id=XFDI4LBNES3M7O4JMVS2YKT6GC2GNPZQSZTZUR4JPADY5YT6GBVA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.000651575 (491h23m21.87173462s) [noise=true , address=145.79.227.38:4978 , piece_id=VBN2SR5SLR53RTA2656K37DUSLJTXYUB65GTXJMJULRA6QTSJRRQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001262057 (491h23m21.872345092s) [noise=true , address=145.79.227.47:4266 , piece_id=ZOOIQO6YLNERNTNOFDYJKEQW4WJXSLZNQGPQ2UVPHP635WRPZZPQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.001831802 (491h23m21.872914857s) [noise=true , address=45.91.116.5:4300 , piece_id=EXZ3LXWBS7M4EQ6JXW25HMVDJUPXDXTNHCRARCXYWP7BLAEEUUCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.012552446 (491h23m21.883635481s) [noise=true , address=145.79.227.11:4732 , piece_id=TH3MB2SK3ZCAN6LZESDADP3Q3PNOYVZD2IMJI552EOWECLAJNCCQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.019006005 (491h23m21.890089051s) [noise=true , address=45.91.116.12:4225 , piece_id=NTAU5WQZUSV23NYWBZTESVMAPN7LXZDKNZTENZ4FZFKBKJKJZXFQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.024025072 (491h23m21.895108107s) [noise=true , address=45.91.116.4:4937 , piece_id=IWV72MZVG24NCSWFWIFRYJO2XW6LOYCXMPDCQU7HV7LJI2LHWXMQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.026746474 (491h23m21.897829519s) [noise=true , address=45.91.116.9:4263 , piece_id=OOYG7BEKKFLXZYHVCO3BUQYWBAX3M5OLVVPUTGJZWFN4IQEIJFYQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:22.030044014 (491h23m21.901127049s) [noise=true , address=145.79.227.52:4447 , piece_id=MHYPNPUAZUZKGVNYTTBCEV5D5K24OJNGM7SACGRW7AVDEGDVBMKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:23.641640782 (491h23m23.512723817s) [noise=true , address=145.79.227.4:4186 , piece_id=HQFXVAVEP4JSM3MDTECDQKN6BGCSULSBUHWQY3OG37GLICMXTZNA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:24.574236128 (491h23m24.445319163s) [noise=true , address=145.79.227.41:4085 , piece_id=UDORIZDSIGCDPXNQ7DYHNA5PJX2YW5WBGRXCMDI2CHD6JOM6VVUQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: <nil> Dec 26 02:52:49.051722782 (491h23m48.922805847s) [noise=true , address=145.79.227.37:4567 , piece_id=ABSMW67JXEK5TKMLYOREKDH2AIIBO23H3MS34Z3A7AUCO6D54KZA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.37:4567: use of closed network connection Dec 26 02:52:49.051839014 (491h23m48.922922059s) [noise=true , address=145.79.227.13:4802 , piece_id=UPDP3K44OTSH6PTK2XUWATGWMUSM3PQ722LWTFBM3H7F2BKMVCKA, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58960->145.79.227.13:4802: use of closed network connection Dec 26 02:52:49.051985743 (491h23m48.923068778s) [noise=true , address=145.79.227.35:4482 , piece_id=KWCK6NZJDMFX4EN5DGRRACAIE3LOQQZUPUEFFO3D3ULFQG3RXL4Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: read tcp 188.166.225.170:34318->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052405231 (491h23m48.923488297s) [noise=true , address=145.79.227.49:4338 , piece_id=5PNGITVRGCAOJFLNWEPPJ5M2VM6X5MIC4Y4MI7IBIKIDL6MCIB7Q, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:39244->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052457040 (491h23m48.923540075s) [noise=true , address=145.79.227.16:4910 , piece_id=ZC2YDXPUBZZJYSCT6Y5FMKOUKHQQQO25PLXLWDMVNIPPT4IRHQBQ, seg_pos={0 0}, upload=431272]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:45528->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:52:49.052501265 (491h23m48.92358431s) [seg_pos={0 0}, upload=431272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 26 02:52:49.286483095 (491h23m49.15756613s) [upload=431272]: Done uploading segments... Dec 26 02:52:49.286552166 (491h23m49.157635201s) [upload=431272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.237176396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.157045009s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:52:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 7.157070797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.611875926s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.611893359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.000592584s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 6.000623714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.629894891s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:51:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.629927432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.455440263s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.45547016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.685401692s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:50:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.685434915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.788985842s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.789021839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.535773616s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.535801258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.776547763s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:49:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.776581017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.592122196s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:48:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.5921607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................X.........................................] duration: 5.855933291s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.855973357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................X........................................] duration: 5.789250984s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.789296762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.52192152s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:47:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.521948692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................X.......................................] duration: 5.761920123s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:46:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.761948617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................X......................................] duration: 5.694314874s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:46:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.69440622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.66734814s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.667376374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.571391026s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:45:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.571418739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................X.....................................] duration: 5.805613971s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:45:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.805736193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................X....................................] duration: 5.84958004s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.849607262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.015538618s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:44:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.01556536s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)