Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................XX................X......XXX.] duration: 3.583572198s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:41:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58360433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...................X...] duration: 31.97879171s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 31.978819413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................................XX................X......XXX] duration: 5.002788055s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:41:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002812572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.] duration: 14.315144171s trace: https://jaeger.dev.storj.io/trace/5d868317ce787ff0 FAIL (2025-12-29 03:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA) context deadline exceeded ======LOGS====== 29 03:39:25.931217811 (564h10m25.802300846s) [upload=494645]: Starting upload Dec 29 03:39:25.931273377 (564h10m25.802356412s) [upload=494645]: Done starting upload Dec 29 03:39:25.931469391 (564h10m25.802552426s) [upload=494645]: Uploading segments... Dec 29 03:39:25.931711250 (564h10m25.802794296s) [upload=494645]: Got next segment. Inline: false Dec 29 03:39:25.931762458 (564h10m25.802845503s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:39:25.931769441 (564h10m25.802852897s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:39:26.174866223 (564h10m26.045949258s) [seg_pos={0 0}, upload=494645]: Begin upload segment... Dec 29 03:39:26.174911159 (564h10m26.045994204s) [seg_pos={0 0}, upload=494645]: Done begin upload segment. Dec 29 03:39:26.174922570 (564h10m26.046005615s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.174937790 (564h10m26.046020835s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175624657 (564h10m26.046707692s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175941531 (564h10m26.047024566s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176222386 (564h10m26.047305421s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176509432 (564h10m26.047592477s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176782362 (564h10m26.047865397s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177049660 (564h10m26.048132705s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177387153 (564h10m26.048470188s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177834585 (564h10m26.04891763s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.178904823 (564h10m26.049987859s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.179245491 (564h10m26.050328526s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180029956 (564h10m26.051112991s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180527964 (564h10m26.051610999s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.181227546 (564h10m26.052310591s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.182419296 (564h10m26.053502341s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:27.818937762 (564h10m27.690020797s) [upload=494645]: Next returned nil segment Dec 29 03:39:27.818949433 (564h10m27.690032469s) [upload=494645]: Waiting for error group managing segments... Dec 29 03:39:28.159200710 (564h10m28.030283755s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.162010190 (564h10m28.033093235s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.169628477 (564h10m28.040711512s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170299454 (564h10m28.0413825s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170570491 (564h10m28.041653536s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.193625089 (564h10m28.064708134s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.200961199 (564h10m28.072044244s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.206186509 (564h10m28.077269544s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.350158362 (564h10m28.221241397s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.509542051 (564h10m28.380625086s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.521022572 (564h10m28.392105617s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.541207186 (564h10m28.412290231s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618079111 (564h10m28.489162146s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618105992 (564h10m28.489189027s) [seg_pos={0 0}, upload=494645]: Segment reached optimal threshold of 13 pieces. Dec 29 03:39:28.618285093 (564h10m28.489368128s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection Dec 29 03:39:28.618312606 (564h10m28.489395641s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.618418247 (564h10m28.489501292s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.618423216 (564h10m28.489506251s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619018039 (564h10m28.490101094s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.619037747 (564h10m28.490120782s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619065129 (564h10m28.490148164s) [seg_pos={0 0}, upload=494645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:39:28.619234201 (564h10m28.490317246s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:39:28.619241856 (564h10m28.490324891s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:39:28.904311268 (564h10m28.775394483s) [upload=494645]: Done uploading segments... Dec 29 03:39:28.904330285 (564h10m28.77541332s) [upload=494645]: Upload finished. err: <nil> Duration: 14.31517007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...................X..] duration: 26.306375961s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 26.306405087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................................XX................X......XX] duration: 5.001694934s trace: https://jaeger.dev.storj.io/trace/6de9f17093e778ca FAIL (2025-12-29 03:40:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001732826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX] duration: 30.003037734s trace: https://jaeger.dev.storj.io/trace/5d868317ce787ff0 FAIL (2025-12-29 03:39:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA) context deadline exceeded ======LOGS====== 29 03:39:25.931217811 (564h10m25.802300846s) [upload=494645]: Starting upload Dec 29 03:39:25.931273377 (564h10m25.802356412s) [upload=494645]: Done starting upload Dec 29 03:39:25.931469391 (564h10m25.802552426s) [upload=494645]: Uploading segments... Dec 29 03:39:25.931711250 (564h10m25.802794296s) [upload=494645]: Got next segment. Inline: false Dec 29 03:39:25.931762458 (564h10m25.802845503s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:39:25.931769441 (564h10m25.802852897s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:39:26.174866223 (564h10m26.045949258s) [seg_pos={0 0}, upload=494645]: Begin upload segment... Dec 29 03:39:26.174911159 (564h10m26.045994204s) [seg_pos={0 0}, upload=494645]: Done begin upload segment. Dec 29 03:39:26.174922570 (564h10m26.046005615s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.174937790 (564h10m26.046020835s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175624657 (564h10m26.046707692s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175941531 (564h10m26.047024566s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176222386 (564h10m26.047305421s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176509432 (564h10m26.047592477s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176782362 (564h10m26.047865397s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177049660 (564h10m26.048132705s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177387153 (564h10m26.048470188s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177834585 (564h10m26.04891763s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.178904823 (564h10m26.049987859s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.179245491 (564h10m26.050328526s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180029956 (564h10m26.051112991s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180527964 (564h10m26.051610999s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.181227546 (564h10m26.052310591s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.182419296 (564h10m26.053502341s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:27.818937762 (564h10m27.690020797s) [upload=494645]: Next returned nil segment Dec 29 03:39:27.818949433 (564h10m27.690032469s) [upload=494645]: Waiting for error group managing segments... Dec 29 03:39:28.159200710 (564h10m28.030283755s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.162010190 (564h10m28.033093235s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.169628477 (564h10m28.040711512s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170299454 (564h10m28.0413825s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170570491 (564h10m28.041653536s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.193625089 (564h10m28.064708134s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.200961199 (564h10m28.072044244s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.206186509 (564h10m28.077269544s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.350158362 (564h10m28.221241397s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.509542051 (564h10m28.380625086s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.521022572 (564h10m28.392105617s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.541207186 (564h10m28.412290231s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618079111 (564h10m28.489162146s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618105992 (564h10m28.489189027s) [seg_pos={0 0}, upload=494645]: Segment reached optimal threshold of 13 pieces. Dec 29 03:39:28.618285093 (564h10m28.489368128s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection Dec 29 03:39:28.618312606 (564h10m28.489395641s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.618418247 (564h10m28.489501292s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.618423216 (564h10m28.489506251s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619018039 (564h10m28.490101094s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.619037747 (564h10m28.490120782s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619065129 (564h10m28.490148164s) [seg_pos={0 0}, upload=494645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:39:28.619234201 (564h10m28.490317246s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:39:28.619241856 (564h10m28.490324891s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:39:28.904311268 (564h10m28.775394483s) [upload=494645]: Done uploading segments... Dec 29 03:39:28.904330285 (564h10m28.77541332s) [upload=494645]: Upload finished. err: <nil> Duration: 30.00306125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...................X.] duration: 33.217277208s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:39:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 33.2172943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................XX................X......X] duration: 5.00121498s trace: https://jaeger.dev.storj.io/trace/3d6d508c8a6e1706 FAIL (2025-12-29 03:39:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001238365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXX] duration: 30.006625607s trace: https://jaeger.dev.storj.io/trace/2222aedb10f98fd1 FAIL (2025-12-29 03:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:38:33.809284564 (564h9m33.680367609s) [upload=494634]: Starting upload Dec 29 03:38:33.809362963 (564h9m33.680446008s) [upload=494634]: Done starting upload Dec 29 03:38:33.809670028 (564h9m33.680753073s) [upload=494634]: Uploading segments... Dec 29 03:38:33.809725383 (564h9m33.680808439s) [upload=494634]: Got next segment. Inline: false Dec 29 03:38:33.809763035 (564h9m33.68084608s) [upload=494634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:38:33.809771421 (564h9m33.680854456s) [upload=494634]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:38:34.053876813 (564h9m33.924959858s) [seg_pos={0 0}, upload=494634]: Begin upload segment... Dec 29 03:38:34.053950804 (564h9m33.925033839s) [seg_pos={0 0}, upload=494634]: Done begin upload segment. Dec 29 03:38:34.054010989 (564h9m33.925094034s) [noise=true , address=77.90.31.12:4589 , piece_id=YWAHOSO2L2EXSQS4E4JXQTDD3TI7ZW2ZAP2YFCX4EMOVJ3RU3A2Q, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.054038201 (564h9m33.925121236s) [noise=true , address=145.79.227.2:4113 , piece_id=YDNDGOEIDL4CHZHHW3AMIF7YEAE5VBHJPD7G7S7EYOG7PM3NUYMQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.054747722 (564h9m33.925830767s) [noise=true , address=145.79.227.6:4259 , piece_id=KMS73EGS3X7ZGRFFWHWXWK3LVIAZFUXHXUSJYTJTLNK5M7GCKTDA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.055149867 (564h9m33.926232912s) [noise=true , address=145.79.227.15:4875 , piece_id=URICDEQAC3SXVFHZQTHNLKAWASD7YGPZ6JCGB7Z6SVLJD46CP22Q, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.056308153 (564h9m33.927391188s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.056813445 (564h9m33.92789649s) [noise=true , address=77.90.31.15:4697 , piece_id=E5NB5UKU3HLATM7U64IRNAWY5IFYCGQIYPZHO3OCQCD2SZI647RQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.058047615 (564h9m33.92913066s) [noise=true , address=145.79.227.26:4654 , piece_id=7L55HBYDNGUN5TBW7DKL4MKXBRPFLH4OXFMJ4OVALDXLI5CHRK4A, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.058446965 (564h9m33.92953001s) [noise=true , address=77.90.31.19:4842 , piece_id=6GRC77URCWUVLLW7DMNF6XCLCAY5QCR7HSOBCBSMFL5JGT5QUNDQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.059606042 (564h9m33.930689077s) [noise=true , address=145.79.227.47:4266 , piece_id=EN5UO2TJV623WLQGECGLYBXNOAK2GDCDMRTYQQAYKWC4YCEEXWSQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.060050579 (564h9m33.931133634s) [noise=true , address=145.79.227.41:4087 , piece_id=TJUOPLUCCMOF7E7QLF6E2ICV5FO2WSY4CLOIZR5CCQMDEBXFHQVA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.061691734 (564h9m33.932774779s) [noise=true , address=77.90.31.14:4662 , piece_id=7NHO2J25S72AOOMG7PBRW4VA3JJPZSRPWY5BAAOF3P5H7GHDLIDQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.063780040 (564h9m33.934863085s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.065939432 (564h9m33.937022477s) [noise=true , address=45.91.116.13:4155 , piece_id=GYDHPXFNUQ2DLWSEACXYPWTU35C7E5YOEK6ISLGFSQHOROWYFOTQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.067095885 (564h9m33.93817892s) [noise=true , address=145.79.227.19:4077 , piece_id=UQWQJ3KBIW3QE2HCKKSTQIM4UIU2Q26JNUHO2XQJEJIJDDBEET6A, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.070897824 (564h9m33.941980859s) [noise=true , address=145.79.227.11:4731 , piece_id=2LVLAHRAN64CH4UT7QMWZR5QEGC7CWNRPLFCSSIAUQ2BJVIRNAMA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.071753473 (564h9m33.942836518s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:35.749565138 (564h9m35.620648173s) [upload=494634]: Next returned nil segment Dec 29 03:38:35.749578173 (564h9m35.620661209s) [upload=494634]: Waiting for error group managing segments... Dec 29 03:38:36.048250248 (564h9m35.919333283s) [noise=true , address=145.79.227.41:4087 , piece_id=TJUOPLUCCMOF7E7QLF6E2ICV5FO2WSY4CLOIZR5CCQMDEBXFHQVA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.050822795 (564h9m35.921905832s) [noise=true , address=145.79.227.19:4077 , piece_id=UQWQJ3KBIW3QE2HCKKSTQIM4UIU2Q26JNUHO2XQJEJIJDDBEET6A, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.051567534 (564h9m35.922650579s) [noise=true , address=145.79.227.11:4731 , piece_id=2LVLAHRAN64CH4UT7QMWZR5QEGC7CWNRPLFCSSIAUQ2BJVIRNAMA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.052444304 (564h9m35.923527339s) [noise=true , address=145.79.227.15:4875 , piece_id=URICDEQAC3SXVFHZQTHNLKAWASD7YGPZ6JCGB7Z6SVLJD46CP22Q, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.058063754 (564h9m35.92914679s) [noise=true , address=145.79.227.2:4113 , piece_id=YDNDGOEIDL4CHZHHW3AMIF7YEAE5VBHJPD7G7S7EYOG7PM3NUYMQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.070523619 (564h9m35.941606654s) [noise=true , address=145.79.227.26:4654 , piece_id=7L55HBYDNGUN5TBW7DKL4MKXBRPFLH4OXFMJ4OVALDXLI5CHRK4A, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.085661433 (564h9m35.956744468s) [noise=true , address=145.79.227.47:4266 , piece_id=EN5UO2TJV623WLQGECGLYBXNOAK2GDCDMRTYQQAYKWC4YCEEXWSQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.100766075 (564h9m35.97184911s) [noise=true , address=145.79.227.6:4259 , piece_id=KMS73EGS3X7ZGRFFWHWXWK3LVIAZFUXHXUSJYTJTLNK5M7GCKTDA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.243856400 (564h9m36.114939435s) [noise=true , address=77.90.31.15:4697 , piece_id=E5NB5UKU3HLATM7U64IRNAWY5IFYCGQIYPZHO3OCQCD2SZI647RQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.369044977 (564h9m36.240128022s) [noise=true , address=77.90.31.19:4842 , piece_id=6GRC77URCWUVLLW7DMNF6XCLCAY5QCR7HSOBCBSMFL5JGT5QUNDQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.429364744 (564h9m36.300447779s) [noise=true , address=77.90.31.12:4589 , piece_id=YWAHOSO2L2EXSQS4E4JXQTDD3TI7ZW2ZAP2YFCX4EMOVJ3RU3A2Q, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.438569582 (564h9m36.309652627s) [noise=true , address=77.90.31.14:4662 , piece_id=7NHO2J25S72AOOMG7PBRW4VA3JJPZSRPWY5BAAOF3P5H7GHDLIDQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.459456063 (564h9m36.330539098s) [noise=true , address=45.91.116.13:4155 , piece_id=GYDHPXFNUQ2DLWSEACXYPWTU35C7E5YOEK6ISLGFSQHOROWYFOTQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.459476832 (564h9m36.330559867s) [seg_pos={0 0}, upload=494634]: Segment reached optimal threshold of 13 pieces. Dec 29 03:38:36.459671734 (564h9m36.330754769s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection Dec 29 03:38:36.459679789 (564h9m36.330762824s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459692293 (564h9m36.330775328s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 03:38:36.459700709 (564h9m36.330783744s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459874279 (564h9m36.330957314s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 03:38:36.459888777 (564h9m36.330971812s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459915488 (564h9m36.330998523s) [seg_pos={0 0}, upload=494634]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:38:36.460008996 (564h9m36.331092031s) [upload=494634]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:38:36.460014246 (564h9m36.331097281s) [upload=494634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:38:36.749698446 (564h9m36.620781491s) [upload=494634]: Done uploading segments... Dec 29 03:38:36.749721269 (564h9m36.620804314s) [upload=494634]: Upload finished. err: <nil> Duration: 30.006652468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XX................X......] duration: 3.516119895s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:38:43 UTC) orbiter warning: context deadline exceeded Duration: 3.516147709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 1m0.002790925s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 1m0.002812336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXX] duration: 30.003831039s trace: https://jaeger.dev.storj.io/trace/6218c6328ad159ba FAIL (2025-12-29 03:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA) context deadline exceeded ======LOGS====== Dec 29 03:37:51.775433780 (564h8m51.646516815s) [upload=494624]: Starting upload Dec 29 03:37:51.775494756 (564h8m51.646577801s) [upload=494624]: Done starting upload Dec 29 03:37:51.775678457 (564h8m51.646761502s) [upload=494624]: Uploading segments... Dec 29 03:37:51.775778055 (564h8m51.6468611s) [upload=494624]: Got next segment. Inline: false Dec 29 03:37:51.775834684 (564h8m51.646917719s) [upload=494624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:37:51.775841116 (564h8m51.646924171s) [upload=494624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:37:52.030321923 (564h8m51.901404988s) [seg_pos={0 0}, upload=494624]: Begin upload segment... Dec 29 03:37:52.030376186 (564h8m51.901459231s) [seg_pos={0 0}, upload=494624]: Done begin upload segment. Dec 29 03:37:52.030404280 (564h8m51.901487315s) [noise=true , address=145.79.227.11:4731 , piece_id=6IXJSF6ENUMTTDUJCT4AK5SE26YV22KC6DFSDKHKEI2FEIANJJMA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.030434828 (564h8m51.901517863s) [noise=true , address=145.79.227.16:4909 , piece_id=36MFSEXLM3GDHKUYY2B4WR4YEB5TDNI34HOWHGX64SNIJQ3O46EQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.030950499 (564h8m51.902033545s) [noise=true , address=45.91.116.10:4082 , piece_id=TUBB4WCM5V7R53AHAZR4U4YRF2T7F47VY3TI6JSNIPIIMA445N5A, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031224902 (564h8m51.902307947s) [noise=true , address=145.79.227.13:4803 , piece_id=HJBBYFPW3OVFJEFPAOFGXZLW6DQCRCK4DEZQNXLCGKFLPY5RCXSQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031261311 (564h8m51.902344346s) [noise=true , address=77.90.31.14:4661 , piece_id=K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031576151 (564h8m51.902659186s) [noise=true , address=45.91.116.5:4299 , piece_id=CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031799677 (564h8m51.902882712s) [noise=true , address=45.91.116.6:4976 , piece_id=CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031865042 (564h8m51.902948087s) [noise=true , address=145.79.227.51:4410 , piece_id=ZZWLHJEDPV5KVKPRNJYKFW4TBRD7INYTA4JU24XTMY3RFBXGCQCQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032140787 (564h8m51.903223822s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032259763 (564h8m51.903342798s) [noise=true , address=145.79.227.17:4972 , piece_id=WUDUWG4OAZULVRWEWKAZS62PDYHZHU22BHN3SZPJQRTPUG6QKWUQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032531400 (564h8m51.903614435s) [noise=true , address=145.79.227.48:4302 , piece_id=NZSKSFIBXVZK5G5QYRRVFMHOEVVQ3ILLLJL5NRIOUVUITSMRRDWA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032630509 (564h8m51.903713544s) [noise=true , address=145.79.227.15:4873 , piece_id=3GPQIKFAUJZJJPCHF7CFZ5RNNSSQUCT4KM3RRZCFRYWJU63OH6RQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032851760 (564h8m51.903934805s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032919900 (564h8m51.904002935s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.033237264 (564h8m51.904320309s) [noise=true , address=145.79.227.26:4653 , piece_id=MIY4VQOMEJES44DKGJDNTWV6VG6O5XV625AZ5XTWTHKQOKVBBY2Q, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.033674457 (564h8m51.904757492s) [noise=true , address=145.79.227.21:4473 , piece_id=IGIZ6DRB7LFMXG2MYAJUMZBMGZ6BRABT6637WZM6KXFWGUS42QIQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:53.730319692 (564h8m53.601402728s) [upload=494624]: Next returned nil segment Dec 29 03:37:53.730332837 (564h8m53.601415872s) [upload=494624]: Waiting for error group managing segments... Dec 29 03:37:54.002225981 (564h8m53.873309046s) [noise=true , address=145.79.227.15:4873 , piece_id=3GPQIKFAUJZJJPCHF7CFZ5RNNSSQUCT4KM3RRZCFRYWJU63OH6RQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.002763665 (564h8m53.873846721s) [noise=true , address=145.79.227.11:4731 , piece_id=6IXJSF6ENUMTTDUJCT4AK5SE26YV22KC6DFSDKHKEI2FEIANJJMA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.014420301 (564h8m53.885503336s) [noise=true , address=145.79.227.48:4302 , piece_id=NZSKSFIBXVZK5G5QYRRVFMHOEVVQ3ILLLJL5NRIOUVUITSMRRDWA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.023837253 (564h8m53.894920298s) [noise=true , address=145.79.227.17:4972 , piece_id=WUDUWG4OAZULVRWEWKAZS62PDYHZHU22BHN3SZPJQRTPUG6QKWUQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.032246635 (564h8m53.90332967s) [noise=true , address=145.79.227.26:4653 , piece_id=MIY4VQOMEJES44DKGJDNTWV6VG6O5XV625AZ5XTWTHKQOKVBBY2Q, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.035564372 (564h8m53.906647417s) [noise=true , address=145.79.227.16:4909 , piece_id=36MFSEXLM3GDHKUYY2B4WR4YEB5TDNI34HOWHGX64SNIJQ3O46EQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.036209321 (564h8m53.907292366s) [noise=true , address=145.79.227.51:4410 , piece_id=ZZWLHJEDPV5KVKPRNJYKFW4TBRD7INYTA4JU24XTMY3RFBXGCQCQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.037539273 (564h8m53.908622308s) [noise=true , address=145.79.227.13:4803 , piece_id=HJBBYFPW3OVFJEFPAOFGXZLW6DQCRCK4DEZQNXLCGKFLPY5RCXSQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.038018055 (564h8m53.90910109s) [noise=true , address=145.79.227.21:4473 , piece_id=IGIZ6DRB7LFMXG2MYAJUMZBMGZ6BRABT6637WZM6KXFWGUS42QIQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.278062964 (564h8m54.149146009s) [noise=true , address=45.91.116.5:4299 , piece_id=CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.280496568 (564h8m54.151579614s) [noise=true , address=45.91.116.10:4082 , piece_id=TUBB4WCM5V7R53AHAZR4U4YRF2T7F47VY3TI6JSNIPIIMA445N5A, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.335981459 (564h8m54.207064504s) [noise=true , address=45.91.116.6:4976 , piece_id=CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.924013146 (564h8m54.795096181s) [noise=true , address=77.90.31.14:4661 , piece_id=K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.924031430 (564h8m54.795114475s) [seg_pos={0 0}, upload=494624]: Segment reached optimal threshold of 13 pieces. Dec 29 03:37:54.924187348 (564h8m54.795270383s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection Dec 29 03:37:54.924192798 (564h8m54.795275833s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924369064 (564h8m54.795452099s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:54.924375045 (564h8m54.79545808s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924453515 (564h8m54.79553655s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:54.924474374 (564h8m54.795557409s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924508249 (564h8m54.795591284s) [seg_pos={0 0}, upload=494624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:37:54.924631043 (564h8m54.795714078s) [upload=494624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:54.924636243 (564h8m54.795719278s) [upload=494624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:55.216746082 (564h8m55.087829137s) [upload=494624]: Done uploading segments... Dec 29 03:37:55.216827677 (564h8m55.087910712s) [upload=494624]: Upload finished. err: <nil> Duration: 30.003865835s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XX................X.....] duration: 4.625654297s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:37:59 UTC) orbiter warning: context deadline exceeded Duration: 4.625673684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 56.232845346s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:37:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 56.232869192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXX] duration: 30.003147959s trace: https://jaeger.dev.storj.io/trace/1411f8ba3ffad9d5 FAIL (2025-12-29 03:37:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ) context deadline exceeded ======LOGS====== 08]: Starting upload Dec 29 03:36:49.301462682 (564h7m49.172545717s) [upload=494608]: Done starting upload Dec 29 03:36:49.301526193 (564h7m49.172609228s) [upload=494608]: Uploading segments... Dec 29 03:36:49.301688091 (564h7m49.172771126s) [upload=494608]: Got next segment. Inline: false Dec 29 03:36:49.301752564 (564h7m49.172835609s) [upload=494608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:36:49.301760018 (564h7m49.172843113s) [upload=494608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:36:49.547947304 (564h7m49.419030339s) [seg_pos={0 0}, upload=494608]: Begin upload segment... Dec 29 03:36:49.548014071 (564h7m49.419097116s) [seg_pos={0 0}, upload=494608]: Done begin upload segment. Dec 29 03:36:49.548105435 (564h7m49.419188481s) [noise=true , address=145.79.227.36:4517 , piece_id=RYDCIQPCI2ADU3MQGP3EG6QJKQU2OZ74SYNLRU75ZVSDX3G46WOQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548123489 (564h7m49.419206524s) [noise=true , address=145.79.227.39:4015 , piece_id=HCHKGBMSYMR5JCL5YNYCGWX7U3UYFRZH6N3D2BUVLPP5Y35LADFQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548188363 (564h7m49.419271398s) [noise=true , address=145.79.227.37:4567 , piece_id=VQU6DWRUHYWVYDKFK3WNAKABDJQTSKCZXCHB2U2OM6E3V3SCD52A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548825607 (564h7m49.419908642s) [noise=true , address=145.79.227.2:4113 , piece_id=METUMZTP3SQYMS4ZUSXVQIE6Q5L6C7UEJTCSTQQMXEN2246LQ4VA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.549249264 (564h7m49.420332299s) [noise=true , address=45.91.116.5:4298 , piece_id=SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.549679062 (564h7m49.420762107s) [noise=true , address=45.91.116.9:4262 , piece_id=2YMSN24ST7AHPJWOF3JGPS4W4HWSXLGP7HWDPGSIKJ3OVFJWOHBQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.550209101 (564h7m49.421292146s) [noise=true , address=45.91.116.10:4083 , piece_id=KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.550729522 (564h7m49.421812557s) [noise=true , address=77.90.31.11:4554 , piece_id=YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.551239493 (564h7m49.422322538s) [noise=true , address=145.79.227.46:4601 , piece_id=FRO6WKYZO565BATDK2EI2BEFNHOJUOOSCE2ZQ6EJ2D3PYRXVQR2A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.551720660 (564h7m49.422803695s) [noise=true , address=145.79.227.45:4230 , piece_id=TNLHZXR4KNWTEKCMN5YRD3TEECDIVZMBTEIMPMNDDWNDQQVV6AOA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.552210913 (564h7m49.423293958s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.553436116 (564h7m49.424519161s) [noise=true , address=145.79.227.32:4835 , piece_id=5J7NKVRPAN7CCPMC4RAAWH62Z3OF3QZIOOMP3JI6M6QHZHIK6TPA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.553744994 (564h7m49.424828039s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.555323159 (564h7m49.426406194s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.555468135 (564h7m49.42655117s) [noise=true , address=145.79.227.11:4732 , piece_id=UXLX7JE23UR27CUSBQCXWKLVVDNON7MPTLYQQXE7UYQ2X6SF2AJA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.557144577 (564h7m49.428227622s) [noise=true , address=145.79.227.13:4802 , piece_id=6ADJZ2RTSJLXU2KPZ5UPGNMJCFFMQL5O2AWKKUC2JBK6HSC2IRXA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:51.254589086 (564h7m51.125672131s) [upload=494608]: Next returned nil segment Dec 29 03:36:51.254602902 (564h7m51.125685937s) [upload=494608]: Waiting for error group managing segments... Dec 29 03:36:51.535758903 (564h7m51.406841938s) [noise=true , address=145.79.227.37:4567 , piece_id=VQU6DWRUHYWVYDKFK3WNAKABDJQTSKCZXCHB2U2OM6E3V3SCD52A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.557049213 (564h7m51.428132248s) [noise=true , address=145.79.227.45:4230 , piece_id=TNLHZXR4KNWTEKCMN5YRD3TEECDIVZMBTEIMPMNDDWNDQQVV6AOA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.557108195 (564h7m51.428191241s) [noise=true , address=145.79.227.32:4835 , piece_id=5J7NKVRPAN7CCPMC4RAAWH62Z3OF3QZIOOMP3JI6M6QHZHIK6TPA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.568152976 (564h7m51.439236021s) [noise=true , address=145.79.227.13:4802 , piece_id=6ADJZ2RTSJLXU2KPZ5UPGNMJCFFMQL5O2AWKKUC2JBK6HSC2IRXA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.573632841 (564h7m51.444715886s) [noise=true , address=145.79.227.39:4015 , piece_id=HCHKGBMSYMR5JCL5YNYCGWX7U3UYFRZH6N3D2BUVLPP5Y35LADFQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.575501148 (564h7m51.446584194s) [noise=true , address=145.79.227.46:4601 , piece_id=FRO6WKYZO565BATDK2EI2BEFNHOJUOOSCE2ZQ6EJ2D3PYRXVQR2A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.576209948 (564h7m51.447292983s) [noise=true , address=145.79.227.36:4517 , piece_id=RYDCIQPCI2ADU3MQGP3EG6QJKQU2OZ74SYNLRU75ZVSDX3G46WOQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.597716239 (564h7m51.468799274s) [noise=true , address=145.79.227.11:4732 , piece_id=UXLX7JE23UR27CUSBQCXWKLVVDNON7MPTLYQQXE7UYQ2X6SF2AJA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.863055223 (564h7m51.734138258s) [noise=true , address=45.91.116.10:4083 , piece_id=KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.924048513 (564h7m51.795131548s) [noise=true , address=145.79.227.2:4113 , piece_id=METUMZTP3SQYMS4ZUSXVQIE6Q5L6C7UEJTCSTQQMXEN2246LQ4VA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.952469681 (564h7m51.823552716s) [noise=true , address=45.91.116.5:4298 , piece_id=SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.116013461 (564h7m51.987096506s) [noise=true , address=77.90.31.11:4554 , piece_id=YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.267267800 (564h7m52.138350835s) [noise=true , address=45.91.116.9:4262 , piece_id=2YMSN24ST7AHPJWOF3JGPS4W4HWSXLGP7HWDPGSIKJ3OVFJWOHBQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.267289492 (564h7m52.138372527s) [seg_pos={0 0}, upload=494608]: Segment reached optimal threshold of 13 pieces. Dec 29 03:36:52.267485836 (564h7m52.138568871s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection Dec 29 03:36:52.267491437 (564h7m52.138574492s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267707478 (564h7m52.138790513s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:39836->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 03:36:52.267713650 (564h7m52.138796685s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267944780 (564h7m52.139027815s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 03:36:52.267964998 (564h7m52.139048033s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267999644 (564h7m52.139082679s) [seg_pos={0 0}, upload=494608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:36:52.268160901 (564h7m52.139243946s) [upload=494608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:36:52.268169918 (564h7m52.139252953s) [upload=494608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:36:52.555582033 (564h7m52.426665068s) [upload=494608]: Done uploading segments... Dec 29 03:36:52.555601900 (564h7m52.426684935s) [upload=494608]: Upload finished. err: <nil> Duration: 30.003172755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XX................X....] duration: 3.739559843s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:36:48 UTC) orbiter warning: context deadline exceeded Duration: 3.739585712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 41.548921606s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 41.548954588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXX] duration: 30.003291504s trace: https://jaeger.dev.storj.io/trace/7ed6cfc30e580aff FAIL (2025-12-29 03:36:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:35:36.275778176 (564h6m36.146861211s) [upload=494591]: Starting upload Dec 29 03:35:36.275854110 (564h6m36.146937156s) [upload=494591]: Done starting upload Dec 29 03:35:36.275899196 (564h6m36.146982231s) [upload=494591]: Uploading segments... Dec 29 03:35:36.276100730 (564h6m36.147183765s) [upload=494591]: Got next segment. Inline: false Dec 29 03:35:36.276151427 (564h6m36.147234462s) [upload=494591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:35:36.276157308 (564h6m36.147240343s) [upload=494591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:35:36.524095518 (564h6m36.395178563s) [seg_pos={0 0}, upload=494591]: Begin upload segment... Dec 29 03:35:36.524146084 (564h6m36.395229119s) [seg_pos={0 0}, upload=494591]: Done begin upload segment. Dec 29 03:35:36.524267115 (564h6m36.39535015s) [noise=true , address=145.79.227.7:4295 , piece_id=XEOLZELPGJ6OFF47SGVKRO7YILSEFLZ7YTVCKB7HQG6BEUVFBQOA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.525352892 (564h6m36.396435927s) [noise=true , address=145.79.227.29:4941 , piece_id=RKC6HCTP7EUWZNHEWXV4UMCW2UQ4UOHHYHPYIJLULR2J4QO5OCJA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.525841132 (564h6m36.396924167s) [noise=true , address=77.90.31.19:4842 , piece_id=HVUH5LTV643N4DAURS5ZR4UZGC5UFQ4QT3EEPRZDOHGUPUQIAJWA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.526201497 (564h6m36.397284533s) [noise=true , address=45.91.116.9:4264 , piece_id=UPI3VIAOUT7GEEY2VDUFP43XO567PIOTS2MBKRX3NOHLXZDKL44A, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.526377002 (564h6m36.397460047s) [noise=true , address=77.90.31.14:4664 , piece_id=7H466LYA6UXZZO6IEC7VYKVQK3ZI6HOXTIP2DZSHCB3VDDK6Q53Q, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.526557226 (564h6m36.397640271s) [noise=true , address=145.79.227.41:4085 , piece_id=WHD2DH2QZMNJDISNGFKQDIPPDNL6TBZAF3UANJHRLPNHBMEU6THA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.526904547 (564h6m36.397987592s) [noise=true , address=145.79.227.6:4258 , piece_id=55NR4KVFHOVZJ77YTWC3MODITVHVJIW4N3ODU3LCU7V42JHIGZSA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.526938522 (564h6m36.398021557s) [noise=true , address=145.79.227.48:4302 , piece_id=ZBYIXXUDRVR4TQMULU67M7AAAHPPRH272UBLFD2R6BBFYGBYT2GA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.527234986 (564h6m36.398318021s) [noise=true , address=45.91.116.6:4973 , piece_id=OATCZN3CIFW5Q6QHXL2625VTMS7B4TQKZJIPU2SQ7HLI2F4KZU7Q, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.527289981 (564h6m36.398373016s) [noise=true , address=145.79.227.17:4971 , piece_id=WSTJB3ZNUEG3NG6L3JOKTJWKFYBMFNMJEWP5HNNMUUJPFTJZI32A, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.527777068 (564h6m36.398860103s) [noise=true , address=77.90.31.10:4517 , piece_id=KUTFXR7RLR24VT5LITC5DK22KGRUTRCIZDY5EM44Z2DG4EF4YDOA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.528221734 (564h6m36.399304769s) [noise=true , address=145.79.227.50:4374 , piece_id=ASL3ORX6CXXZVPGJ4BM7KUCJJ5GLJW5I6X6EOG7NK5A6RC7LTPXA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.528373915 (564h6m36.39945695s) [noise=true , address=145.79.227.3:4150 , piece_id=NAUZJH4BFIQWKBQ3WTI2DVWT2CAQ7QLBNYA24QEB5HQHS2O7Z3ZQ, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.528524281 (564h6m36.399607316s) [noise=true , address=145.79.227.36:4517 , piece_id=GXY5L4T6AQAH7XMXRB27O2ZZMWFLJNFDT3E2EMKP6XJEIW5FWHDQ, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.528702159 (564h6m36.399785194s) [noise=true , address=145.79.227.32:4834 , piece_id=M4GZZNHYM2GE2ZD7FJ5WIYOK4CJQUACFUKOTCJMNAP7EOE6OKIZA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:36.529021958 (564h6m36.400104993s) [noise=true , address=145.79.227.47:4265 , piece_id=K5WRN4DMRECPNNVTF3IHWZIWFDEJIHAFBRCY7MPSJK7P6DSJAZJA, seg_pos={0 0}, upload=494591]: Uploading piece... Dec 29 03:35:38.218195669 (564h6m38.089278714s) [upload=494591]: Next returned nil segment Dec 29 03:35:38.218215788 (564h6m38.089298823s) [upload=494591]: Waiting for error group managing segments... Dec 29 03:35:38.505380229 (564h6m38.376463274s) [noise=true , address=145.79.227.32:4834 , piece_id=M4GZZNHYM2GE2ZD7FJ5WIYOK4CJQUACFUKOTCJMNAP7EOE6OKIZA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.508059401 (564h6m38.379142446s) [noise=true , address=145.79.227.50:4374 , piece_id=ASL3ORX6CXXZVPGJ4BM7KUCJJ5GLJW5I6X6EOG7NK5A6RC7LTPXA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.523383380 (564h6m38.394466415s) [noise=true , address=145.79.227.47:4265 , piece_id=K5WRN4DMRECPNNVTF3IHWZIWFDEJIHAFBRCY7MPSJK7P6DSJAZJA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.527149391 (564h6m38.398232426s) [noise=true , address=145.79.227.6:4258 , piece_id=55NR4KVFHOVZJ77YTWC3MODITVHVJIW4N3ODU3LCU7V42JHIGZSA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.528603841 (564h6m38.399686886s) [noise=true , address=145.79.227.41:4085 , piece_id=WHD2DH2QZMNJDISNGFKQDIPPDNL6TBZAF3UANJHRLPNHBMEU6THA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.528927687 (564h6m38.400010722s) [noise=true , address=145.79.227.17:4971 , piece_id=WSTJB3ZNUEG3NG6L3JOKTJWKFYBMFNMJEWP5HNNMUUJPFTJZI32A, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.533569295 (564h6m38.40465237s) [noise=true , address=145.79.227.7:4295 , piece_id=XEOLZELPGJ6OFF47SGVKRO7YILSEFLZ7YTVCKB7HQG6BEUVFBQOA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.533821155 (564h6m38.4049042s) [noise=true , address=145.79.227.48:4302 , piece_id=ZBYIXXUDRVR4TQMULU67M7AAAHPPRH272UBLFD2R6BBFYGBYT2GA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.558623002 (564h6m38.429706037s) [noise=true , address=145.79.227.3:4150 , piece_id=NAUZJH4BFIQWKBQ3WTI2DVWT2CAQ7QLBNYA24QEB5HQHS2O7Z3ZQ, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.841447700 (564h6m38.712530735s) [noise=true , address=77.90.31.10:4517 , piece_id=KUTFXR7RLR24VT5LITC5DK22KGRUTRCIZDY5EM44Z2DG4EF4YDOA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.860339061 (564h6m38.731422106s) [noise=true , address=77.90.31.14:4664 , piece_id=7H466LYA6UXZZO6IEC7VYKVQK3ZI6HOXTIP2DZSHCB3VDDK6Q53Q, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:38.934533148 (564h6m38.805616183s) [noise=true , address=45.91.116.6:4973 , piece_id=OATCZN3CIFW5Q6QHXL2625VTMS7B4TQKZJIPU2SQ7HLI2F4KZU7Q, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:39.501401667 (564h6m39.372484703s) [noise=true , address=77.90.31.19:4842 , piece_id=HVUH5LTV643N4DAURS5ZR4UZGC5UFQ4QT3EEPRZDOHGUPUQIAJWA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: <nil> Dec 29 03:35:39.501420423 (564h6m39.372503458s) [seg_pos={0 0}, upload=494591]: Segment reached optimal threshold of 13 pieces. Dec 29 03:35:39.501628960 (564h6m39.372712005s) [noise=true , address=45.91.116.9:4264 , piece_id=UPI3VIAOUT7GEEY2VDUFP43XO567PIOTS2MBKRX3NOHLXZDKL44A, seg_pos={0 0}, upload=494591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 03:35:39.501638738 (564h6m39.372721773s) [noise=true , address=45.91.116.9:4264 , piece_id=UPI3VIAOUT7GEEY2VDUFP43XO567PIOTS2MBKRX3NOHLXZDKL44A, seg_pos={0 0}, upload=494591]: Long tail cancellation due to optimal threshold Dec 29 03:35:39.501867424 (564h6m39.372950459s) [noise=true , address=145.79.227.36:4517 , piece_id=GXY5L4T6AQAH7XMXRB27O2ZZMWFLJNFDT3E2EMKP6XJEIW5FWHDQ, seg_pos={0 0}, upload=494591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: write tcp 188.166.225.170:55152->145.79.227.36:4517: use of closed network connection; write tcp 188.166.225.170:55152->145.79.227.36:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55152->145.79.227.36:4517: use of closed network connection Dec 29 03:35:39.501882612 (564h6m39.372965658s) [noise=true , address=145.79.227.36:4517 , piece_id=GXY5L4T6AQAH7XMXRB27O2ZZMWFLJNFDT3E2EMKP6XJEIW5FWHDQ, seg_pos={0 0}, upload=494591]: Long tail cancellation due to optimal threshold Dec 29 03:35:39.501936195 (564h6m39.37301924s) [noise=true , address=145.79.227.29:4941 , piece_id=RKC6HCTP7EUWZNHEWXV4UMCW2UQ4UOHHYHPYIJLULR2J4QO5OCJA, seg_pos={0 0}, upload=494591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 29 03:35:39.501943449 (564h6m39.373026484s) [noise=true , address=145.79.227.29:4941 , piece_id=RKC6HCTP7EUWZNHEWXV4UMCW2UQ4UOHHYHPYIJLULR2J4QO5OCJA, seg_pos={0 0}, upload=494591]: Long tail cancellation due to optimal threshold Dec 29 03:35:39.501963667 (564h6m39.373046703s) [seg_pos={0 0}, upload=494591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:35:39.502028912 (564h6m39.373111957s) [upload=494591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:35:39.502033631 (564h6m39.373116666s) [upload=494591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:35:39.798458206 (564h6m39.669541251s) [upload=494591]: Done uploading segments... Dec 29 03:35:39.798479275 (564h6m39.669562311s) [upload=494591]: Upload finished. err: <nil> Duration: 30.003319417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XX................X...] duration: 3.555748499s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:36:04 UTC) orbiter warning: context deadline exceeded Duration: 3.555773948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 39.823660019s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:35:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.823682983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XX] duration: 30.003704312s trace: https://jaeger.dev.storj.io/trace/427f001070ded283 FAIL (2025-12-29 03:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: I7RDCBADVWWI3TJ6ABBYRTOFBF6A5Q22H6J4Q6NGXSJRWFU3LLBA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 4TMVDKDKBU4BXQMRBB4TVKSA7WSULETIVFQYNXLA4L5BJXWAOVUA) context deadline exceeded ======LOGS====== Dec 29 03:35:04.572120520 (564h6m4.443203565s) [upload=494584]: Starting upload Dec 29 03:35:04.572213496 (564h6m4.443296561s) [upload=494584]: Done starting upload Dec 29 03:35:04.572480244 (564h6m4.443563289s) [upload=494584]: Uploading segments... Dec 29 03:35:04.572561418 (564h6m4.443644484s) [upload=494584]: Got next segment. Inline: false Dec 29 03:35:04.572604211 (564h6m4.443687256s) [upload=494584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:35:04.572610262 (564h6m4.443693317s) [upload=494584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:35:04.821416925 (564h6m4.69249999s) [seg_pos={0 0}, upload=494584]: Begin upload segment... Dec 29 03:35:04.821857393 (564h6m4.692940438s) [seg_pos={0 0}, upload=494584]: Done begin upload segment. Dec 29 03:35:04.821877702 (564h6m4.692960747s) [noise=true , address=77.90.31.14:4663 , piece_id=EDBF4UDQZCX5SJY77S5UPNKCPZ4JQQBZB6OIHENJQ4JFYZ3AS2IQ, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.821914772 (564h6m4.692997827s) [noise=true , address=45.91.116.8:4047 , piece_id=4TMVDKDKBU4BXQMRBB4TVKSA7WSULETIVFQYNXLA4L5BJXWAOVUA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.822869580 (564h6m4.693952625s) [noise=true , address=145.79.227.24:4582 , piece_id=M2MU7SWEQ2MLMGDLND6N4T4QLPX4YXPCDTAOCD25Y64Q23337E2Q, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.823155946 (564h6m4.694238991s) [noise=true , address=145.79.227.44:4194 , piece_id=4PRW4GWNUGHOQ5Z7Y4QGRTCTAIOEFKEILM5KIARAUXKNEDDLQTGA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.823703808 (564h6m4.694786854s) [noise=true , address=145.79.227.38:4977 , piece_id=SBLZYBGTWR6DDMYJFRZINT5HQ3W4BKC556WZNCMXR2VXTCC2MB2A, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.824289794 (564h6m4.695372839s) [noise=true , address=145.79.227.33:4871 , piece_id=HWPVFA5TAM7A3U3X4GOT32KVF2UEWBMH7DJJSKOJISRNGUKDJOFQ, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.825240095 (564h6m4.69632315s) [noise=true , address=145.79.227.51:4409 , piece_id=RGFUWHLOVY7OW3MNPKZGP5RTKMFM7DUDLE4QGD7LAR3SOYOEUHHQ, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.825773951 (564h6m4.696856996s) [noise=true , address=145.79.227.14:4840 , piece_id=MEND6XHLZUEBQJ7B6TZKUAYA5FA6UUP4NQ26J6FS564NJWHK6NNA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.826295173 (564h6m4.697378218s) [noise=true , address=145.79.227.17:4970 , piece_id=KNTJUFPTSJ2WNJRTJ7HIJ44WI7ZA5NW54ARXANICJ655AK3WSR5Q, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.826776891 (564h6m4.697859936s) [noise=true , address=145.79.227.15:4873 , piece_id=XMI7BITB4QHI7EDJJTOOCM5ELGAYZ2B6TKFEJ4QPBXL6FB5QPSDQ, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.827255542 (564h6m4.698338587s) [noise=true , address=145.79.227.46:4602 , piece_id=B3ARXBTN7J7URD7TGHFQID3E3F4SSTLECZERIOJQICTO66J7DCLA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.827905790 (564h6m4.698988835s) [noise=true , address=145.79.227.37:4565 , piece_id=62OIA55YNSYU5OFJXO7ATRNFSGJLJNK3D5VSCWAPWKWNZWCJY56Q, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.828040927 (564h6m4.699123983s) [noise=true , address=145.79.227.40:4050 , piece_id=WTLW77N7DRCXMV77DANOFQKQDEWAMNCZXPFPMQO4WXMOQ7MLFTPQ, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.828739387 (564h6m4.699822432s) [noise=true , address=145.79.227.49:4337 , piece_id=ZD4UZGZPWX7P7TIGDTZF344MMGUWB7ZSQL5SUGTDNODG26F4UNJA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.829504074 (564h6m4.700587119s) [noise=true , address=45.91.116.9:4262 , piece_id=I7RDCBADVWWI3TJ6ABBYRTOFBF6A5Q22H6J4Q6NGXSJRWFU3LLBA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:04.830267177 (564h6m4.701350232s) [noise=true , address=77.90.31.11:4554 , piece_id=GKW4IRFZV4RZHNLRTM7CH23ZOWKISAQXLFXNDXWKLUDLIA7V6JTA, seg_pos={0 0}, upload=494584]: Uploading piece... Dec 29 03:35:06.520609662 (564h6m6.391692707s) [upload=494584]: Next returned nil segment Dec 29 03:35:06.520643736 (564h6m6.391726802s) [upload=494584]: Waiting for error group managing segments... Dec 29 03:35:06.808133077 (564h6m6.679216122s) [noise=true , address=145.79.227.51:4409 , piece_id=RGFUWHLOVY7OW3MNPKZGP5RTKMFM7DUDLE4QGD7LAR3SOYOEUHHQ, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.809732162 (564h6m6.680815207s) [noise=true , address=145.79.227.14:4840 , piece_id=MEND6XHLZUEBQJ7B6TZKUAYA5FA6UUP4NQ26J6FS564NJWHK6NNA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.814128293 (564h6m6.685211338s) [noise=true , address=145.79.227.15:4873 , piece_id=XMI7BITB4QHI7EDJJTOOCM5ELGAYZ2B6TKFEJ4QPBXL6FB5QPSDQ, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.818263006 (564h6m6.689346051s) [noise=true , address=145.79.227.24:4582 , piece_id=M2MU7SWEQ2MLMGDLND6N4T4QLPX4YXPCDTAOCD25Y64Q23337E2Q, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.831139785 (564h6m6.70222284s) [noise=true , address=145.79.227.46:4602 , piece_id=B3ARXBTN7J7URD7TGHFQID3E3F4SSTLECZERIOJQICTO66J7DCLA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.832867575 (564h6m6.70395061s) [noise=true , address=145.79.227.38:4977 , piece_id=SBLZYBGTWR6DDMYJFRZINT5HQ3W4BKC556WZNCMXR2VXTCC2MB2A, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:06.836232153 (564h6m6.707315198s) [noise=true , address=145.79.227.17:4970 , piece_id=KNTJUFPTSJ2WNJRTJ7HIJ44WI7ZA5NW54ARXANICJ655AK3WSR5Q, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:07.080926114 (564h6m6.952009159s) [noise=true , address=45.91.116.8:4047 , piece_id=4TMVDKDKBU4BXQMRBB4TVKSA7WSULETIVFQYNXLA4L5BJXWAOVUA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:07.101989252 (564h6m6.973072297s) [noise=true , address=77.90.31.14:4663 , piece_id=EDBF4UDQZCX5SJY77S5UPNKCPZ4JQQBZB6OIHENJQ4JFYZ3AS2IQ, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:07.208106494 (564h6m7.079189539s) [noise=true , address=45.91.116.9:4262 , piece_id=I7RDCBADVWWI3TJ6ABBYRTOFBF6A5Q22H6J4Q6NGXSJRWFU3LLBA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:08.084771307 (564h6m7.955854352s) [noise=true , address=145.79.227.40:4050 , piece_id=WTLW77N7DRCXMV77DANOFQKQDEWAMNCZXPFPMQO4WXMOQ7MLFTPQ, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:08.106058581 (564h6m7.977141626s) [noise=true , address=145.79.227.44:4194 , piece_id=4PRW4GWNUGHOQ5Z7Y4QGRTCTAIOEFKEILM5KIARAUXKNEDDLQTGA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:08.180255122 (564h6m8.051338167s) [noise=true , address=145.79.227.37:4565 , piece_id=62OIA55YNSYU5OFJXO7ATRNFSGJLJNK3D5VSCWAPWKWNZWCJY56Q, seg_pos={0 0}, upload=494584]: Done uploading piece. err: <nil> Dec 29 03:35:08.180273568 (564h6m8.051356603s) [seg_pos={0 0}, upload=494584]: Segment reached optimal threshold of 13 pieces. Dec 29 03:35:08.180456686 (564h6m8.051539721s) [noise=true , address=145.79.227.33:4871 , piece_id=HWPVFA5TAM7A3U3X4GOT32KVF2UEWBMH7DJJSKOJISRNGUKDJOFQ, seg_pos={0 0}, upload=494584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 03:35:08.180463228 (564h6m8.051546263s) [noise=true , address=145.79.227.33:4871 , piece_id=HWPVFA5TAM7A3U3X4GOT32KVF2UEWBMH7DJJSKOJISRNGUKDJOFQ, seg_pos={0 0}, upload=494584]: Long tail cancellation due to optimal threshold Dec 29 03:35:08.180583778 (564h6m8.051666833s) [noise=true , address=145.79.227.49:4337 , piece_id=ZD4UZGZPWX7P7TIGDTZF344MMGUWB7ZSQL5SUGTDNODG26F4UNJA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 03:35:08.180593987 (564h6m8.051677033s) [noise=true , address=145.79.227.49:4337 , piece_id=ZD4UZGZPWX7P7TIGDTZF344MMGUWB7ZSQL5SUGTDNODG26F4UNJA, seg_pos={0 0}, upload=494584]: Long tail cancellation due to optimal threshold Dec 29 03:35:08.180654102 (564h6m8.051737137s) [noise=true , address=77.90.31.11:4554 , piece_id=GKW4IRFZV4RZHNLRTM7CH23ZOWKISAQXLFXNDXWKLUDLIA7V6JTA, seg_pos={0 0}, upload=494584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 03:35:08.180662468 (564h6m8.051745503s) [noise=true , address=77.90.31.11:4554 , piece_id=GKW4IRFZV4RZHNLRTM7CH23ZOWKISAQXLFXNDXWKLUDLIA7V6JTA, seg_pos={0 0}, upload=494584]: Long tail cancellation due to optimal threshold Dec 29 03:35:08.180683348 (564h6m8.051766393s) [seg_pos={0 0}, upload=494584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:35:08.180787677 (564h6m8.051870712s) [upload=494584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:35:08.180794830 (564h6m8.051877865s) [upload=494584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:35:08.483306597 (564h6m8.354389632s) [upload=494584]: Done uploading segments... Dec 29 03:35:08.483332295 (564h6m8.35441534s) [upload=494584]: Upload finished. err: <nil> Duration: 30.003721264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................XX................X..] duration: 3.694960935s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:35:17 UTC) orbiter warning: context deadline exceeded Duration: 3.694990512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...X] duration: 30.004884745s trace: https://jaeger.dev.storj.io/trace/7d1a50d4e2791d41 FAIL (2025-12-29 03:35:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:34:34.325357564 (564h5m34.196440599s) [upload=494578]: Starting upload Dec 29 03:34:34.325444500 (564h5m34.196527545s) [upload=494578]: Done starting upload Dec 29 03:34:34.325532748 (564h5m34.196615783s) [upload=494578]: Uploading segments... Dec 29 03:34:34.325645382 (564h5m34.196728417s) [upload=494578]: Got next segment. Inline: false Dec 29 03:34:34.325675790 (564h5m34.196758835s) [upload=494578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:34:34.325680208 (564h5m34.196763264s) [upload=494578]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:34:34.568838097 (564h5m34.439921132s) [seg_pos={0 0}, upload=494578]: Begin upload segment... Dec 29 03:34:34.568882732 (564h5m34.439965767s) [seg_pos={0 0}, upload=494578]: Done begin upload segment. Dec 29 03:34:34.568910234 (564h5m34.439993269s) [noise=true , address=145.79.227.47:4265 , piece_id=6VWGICJEZ37K6BBMHG6T5OSTC2NREJQ7VRUSNSFPKT7KJPCTTRQA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.568940752 (564h5m34.440023787s) [noise=true , address=145.79.227.45:4231 , piece_id=WGMN4NIHK6DYDOSRXIE4FZNMHAFQTEWK2WBNZGNA6ZD2GWOAF6QQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.569409785 (564h5m34.44049283s) [noise=true , address=145.79.227.39:4014 , piece_id=3GL5XMLHEFL3QJ4CJ4VKWIANIGW2NWDNBX2X4XZYBND2CT2Z4HWA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.569766766 (564h5m34.440849801s) [noise=true , address=77.90.31.21:4877 , piece_id=75HT5U5EUTIGVXTH3ZRB4UMYWVUGZ4ARC2NJHO3O73XGA44JCDDQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.570215780 (564h5m34.441298815s) [noise=true , address=45.91.116.4:4937 , piece_id=NP3WITMN34DRGNTMCCOD3YT63QLBFX2RACBACBDE3GND5L6CE4EA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.570527944 (564h5m34.441610989s) [noise=true , address=145.79.227.6:4259 , piece_id=JG2VGNCU5KMXNZXZ36AQ5W45QPLIZT3QPW3UT2RNHUF4XHUH3WYA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.570922656 (564h5m34.442005691s) [noise=true , address=45.91.116.8:4048 , piece_id=OF3RSLAJERWUYN2X7ZSJW6AIHYG7JLQJUA7ZMOW3G2ZOMT6RQ3KA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.571443198 (564h5m34.442526233s) [noise=true , address=145.79.227.17:4971 , piece_id=HPKFAMUCF2UOA4LEZA3UBEDOQ3J2ZU5L42ZWPHPGHEVH3EEWODRA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.572526570 (564h5m34.443609605s) [noise=true , address=145.79.227.30:4762 , piece_id=AQH6O5Y26REPH3EYBK4ZTPUZYL43YZOFG2E57AIE2GFRC7SSMVNA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.572622192 (564h5m34.443705227s) [noise=true , address=45.91.116.11:4120 , piece_id=XJYVH2UUUOPRJPRTO5TY6ZVOKGCPU3U3GYG674LOBHH6YNK2PP3Q, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.573779826 (564h5m34.444862861s) [noise=true , address=145.79.227.18:4006 , piece_id=NHNMYFSVZYBBSXT4BJLTHHMI5MY4RIN42ZLQTWMN7BS7Y4BDHKRQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.575270665 (564h5m34.44635371s) [noise=true , address=145.79.227.37:4567 , piece_id=AHA4EYW47SGXOLWH7KKJVBHM52TMKQGNXX6OSD4PGLMLU3QCE2WA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.576228900 (564h5m34.447311945s) [noise=true , address=145.79.227.19:4079 , piece_id=LHL55KHJWHUZZB7NBKPLPGPOPVHLK7DEG37UDUXRDRA4XYNQQKOQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.577686706 (564h5m34.448769741s) [noise=true , address=145.79.227.24:4582 , piece_id=VOINA54XEXY2FQO3XHNFXOVMKZ5UV4G6YJ4BM4U6SCYOK5OOGJZQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.578975540 (564h5m34.450058575s) [noise=true , address=45.91.116.5:4297 , piece_id=47FOJYGUX2LUOXPDCYATYKF76XDM6JPJKKT7A5UASJPHDDXZ7LGQ, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:34.580316052 (564h5m34.451399097s) [noise=true , address=145.79.227.32:4834 , piece_id=RGJMVZA42V4H76ATWBZA2KQEQRSYYQVX63JGXAS4VQRZIIZY7SEA, seg_pos={0 0}, upload=494578]: Uploading piece... Dec 29 03:34:36.269138704 (564h5m36.140221749s) [upload=494578]: Next returned nil segment Dec 29 03:34:36.269155826 (564h5m36.140238871s) [upload=494578]: Waiting for error group managing segments... Dec 29 03:34:36.558449182 (564h5m36.429532217s) [noise=true , address=145.79.227.17:4971 , piece_id=HPKFAMUCF2UOA4LEZA3UBEDOQ3J2ZU5L42ZWPHPGHEVH3EEWODRA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.560945265 (564h5m36.43202831s) [noise=true , address=145.79.227.39:4014 , piece_id=3GL5XMLHEFL3QJ4CJ4VKWIANIGW2NWDNBX2X4XZYBND2CT2Z4HWA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.564616767 (564h5m36.435699802s) [noise=true , address=145.79.227.32:4834 , piece_id=RGJMVZA42V4H76ATWBZA2KQEQRSYYQVX63JGXAS4VQRZIIZY7SEA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.567076700 (564h5m36.438159735s) [noise=true , address=145.79.227.37:4567 , piece_id=AHA4EYW47SGXOLWH7KKJVBHM52TMKQGNXX6OSD4PGLMLU3QCE2WA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.581590738 (564h5m36.452673773s) [noise=true , address=145.79.227.45:4231 , piece_id=WGMN4NIHK6DYDOSRXIE4FZNMHAFQTEWK2WBNZGNA6ZD2GWOAF6QQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.582139373 (564h5m36.453222418s) [noise=true , address=145.79.227.18:4006 , piece_id=NHNMYFSVZYBBSXT4BJLTHHMI5MY4RIN42ZLQTWMN7BS7Y4BDHKRQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.582274569 (564h5m36.453357614s) [noise=true , address=145.79.227.47:4265 , piece_id=6VWGICJEZ37K6BBMHG6T5OSTC2NREJQ7VRUSNSFPKT7KJPCTTRQA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.587344043 (564h5m36.458427078s) [noise=true , address=145.79.227.24:4582 , piece_id=VOINA54XEXY2FQO3XHNFXOVMKZ5UV4G6YJ4BM4U6SCYOK5OOGJZQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.603517008 (564h5m36.474600043s) [noise=true , address=145.79.227.19:4079 , piece_id=LHL55KHJWHUZZB7NBKPLPGPOPVHLK7DEG37UDUXRDRA4XYNQQKOQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.649091358 (564h5m36.520174393s) [noise=true , address=145.79.227.6:4259 , piece_id=JG2VGNCU5KMXNZXZ36AQ5W45QPLIZT3QPW3UT2RNHUF4XHUH3WYA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.802749316 (564h5m36.673832351s) [noise=true , address=45.91.116.5:4297 , piece_id=47FOJYGUX2LUOXPDCYATYKF76XDM6JPJKKT7A5UASJPHDDXZ7LGQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.814772139 (564h5m36.685855174s) [noise=true , address=45.91.116.8:4048 , piece_id=OF3RSLAJERWUYN2X7ZSJW6AIHYG7JLQJUA7ZMOW3G2ZOMT6RQ3KA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.852155011 (564h5m36.723238046s) [noise=true , address=77.90.31.21:4877 , piece_id=75HT5U5EUTIGVXTH3ZRB4UMYWVUGZ4ARC2NJHO3O73XGA44JCDDQ, seg_pos={0 0}, upload=494578]: Done uploading piece. err: <nil> Dec 29 03:34:36.852173556 (564h5m36.723256591s) [seg_pos={0 0}, upload=494578]: Segment reached optimal threshold of 13 pieces. Dec 29 03:34:36.852386341 (564h5m36.723469376s) [noise=true , address=45.91.116.4:4937 , piece_id=NP3WITMN34DRGNTMCCOD3YT63QLBFX2RACBACBDE3GND5L6CE4EA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 03:34:36.852392102 (564h5m36.723475137s) [noise=true , address=45.91.116.4:4937 , piece_id=NP3WITMN34DRGNTMCCOD3YT63QLBFX2RACBACBDE3GND5L6CE4EA, seg_pos={0 0}, upload=494578]: Long tail cancellation due to optimal threshold Dec 29 03:34:36.852504166 (564h5m36.723587201s) [noise=true , address=45.91.116.11:4120 , piece_id=XJYVH2UUUOPRJPRTO5TY6ZVOKGCPU3U3GYG674LOBHH6YNK2PP3Q, seg_pos={0 0}, upload=494578]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 03:34:36.852509165 (564h5m36.723592201s) [noise=true , address=45.91.116.11:4120 , piece_id=XJYVH2UUUOPRJPRTO5TY6ZVOKGCPU3U3GYG674LOBHH6YNK2PP3Q, seg_pos={0 0}, upload=494578]: Long tail cancellation due to optimal threshold Dec 29 03:34:36.852740315 (564h5m36.72382336s) [noise=true , address=145.79.227.30:4762 , piece_id=AQH6O5Y26REPH3EYBK4ZTPUZYL43YZOFG2E57AIE2GFRC7SSMVNA, seg_pos={0 0}, upload=494578]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 03:34:36.852758490 (564h5m36.723841525s) [noise=true , address=145.79.227.30:4762 , piece_id=AQH6O5Y26REPH3EYBK4ZTPUZYL43YZOFG2E57AIE2GFRC7SSMVNA, seg_pos={0 0}, upload=494578]: Long tail cancellation due to optimal threshold Dec 29 03:34:36.852812633 (564h5m36.723895668s) [seg_pos={0 0}, upload=494578]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:34:36.852929035 (564h5m36.72401208s) [upload=494578]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:34:36.852937671 (564h5m36.724020706s) [upload=494578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:34:37.137640265 (564h5m37.008723321s) [upload=494578]: Done uploading segments... Dec 29 03:34:37.137664270 (564h5m37.008747306s) [upload=494578]: Upload finished. err: <nil> Duration: 30.004927295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 46.826563418s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 46.826597252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................XX................X.] duration: 3.557831573s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:34:51 UTC) orbiter warning: context deadline exceeded Duration: 3.557855648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 23.13510659s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:34:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 23.135133061s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................XX................X] duration: 5.000783057s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:34:04 UTC) orbiter warning: context deadline exceeded Duration: 5.000825898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...] duration: 9.263911636s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:33:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 9.263933598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 55.114707686s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:33:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 55.114733684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX..] duration: 5.762727589s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 5.762773776s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 3.456734776s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:33:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45676216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................XX...............] duration: 3.48200628s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:32:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482022762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX.] duration: 6.019191115s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 6.01920894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX] duration: 30.002248056s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 30.002270699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................XX..............] duration: 3.538931174s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:31:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538957365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 24.345994931s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 24.34602066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XX] duration: 30.002911898s trace: https://jaeger.dev.storj.io/trace/3827d7cc75f5ac3d FAIL (2025-12-29 03:31:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ) context deadline exceeded ======LOGS====== Dec 29 03:30:43.238069103 (564h1m43.109152148s) [upload=494519]: Starting upload Dec 29 03:30:43.238140439 (564h1m43.109223474s) [upload=494519]: Done starting upload Dec 29 03:30:43.238346962 (564h1m43.109429997s) [upload=494519]: Uploading segments... Dec 29 03:30:43.238433658 (564h1m43.109516703s) [upload=494519]: Got next segment. Inline: false Dec 29 03:30:43.238466059 (564h1m43.109549095s) [upload=494519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:30:43.238471470 (564h1m43.109554505s) [upload=494519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:30:43.498386049 (564h1m43.369469084s) [seg_pos={0 0}, upload=494519]: Begin upload segment... Dec 29 03:30:43.498436755 (564h1m43.369519791s) [seg_pos={0 0}, upload=494519]: Done begin upload segment. Dec 29 03:30:43.498465971 (564h1m43.369549016s) [noise=true , address=45.91.116.12:4228 , piece_id=XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.499298437 (564h1m43.370381492s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.499757881 (564h1m43.370840916s) [noise=true , address=145.79.227.7:4293 , piece_id=QF76422GKZBM2TEAEAYLFGQEIJITAHXSWKMQQTJOT4M5MCEACZFA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500132034 (564h1m43.371215069s) [noise=true , address=145.79.227.50:4375 , piece_id=4XRC5S3ZLO3V3Y45QOGDLDHWQIFW2AOGIUCKJN4Y7YURFSUK4R4A, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500414532 (564h1m43.371497567s) [noise=true , address=145.79.227.51:4410 , piece_id=N5SEDJHTZZ5VY2TCWKKZXC2TINFLYDIBEEWEZTKZI634XGSI5JOA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500693393 (564h1m43.371776428s) [noise=true , address=145.79.227.48:4301 , piece_id=GI6HQC3ZYOH7LFFUAAPVKAPTX5R2KIOFWAFP4T46I3GOVVY3TEOQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500992082 (564h1m43.372075117s) [noise=true , address=145.79.227.47:4265 , piece_id=272MWUCZEUVQFUE3DFQHLAC6GKOU4KHCLWOYCIB6HENN5YFSK2JA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.501284328 (564h1m43.372367363s) [noise=true , address=77.90.31.19:4841 , piece_id=MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.501594429 (564h1m43.372677464s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.502006103 (564h1m43.373089138s) [noise=true , address=145.79.227.5:4223 , piece_id=TRHTPQH7YGRUHPHECPVKFXR2WRHG7HFEBNKBJVBGSQPZYE3WK2JQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.502868484 (564h1m43.373951519s) [noise=true , address=145.79.227.13:4803 , piece_id=KW7F7VVT6SUQPJKPHNYYAKKJBTRR4EUABTFFDOWAPW54OIV6HWMQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.503269528 (564h1m43.374352573s) [noise=true , address=145.79.227.16:4911 , piece_id=DHHUOKE3J6UIMPXJIDF7XFTJ6AFML4CWGCL7UWFYSTEDLWJK4GWQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.503907402 (564h1m43.374990447s) [noise=true , address=145.79.227.15:4875 , piece_id=CGM4KY3VN4EACJTTL53J234BDEDT7B2JFZPJJQFNKO2XAEMVWEOQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.505050990 (564h1m43.376134035s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.507264085 (564h1m43.37834713s) [noise=true , address=145.79.227.14:4838 , piece_id=2Z6BAR3AM2UPJAUKQ7STBHSCQMXMVJ2XOMHX5W2IA7XZ7GXI4YPA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.508957769 (564h1m43.380040814s) [noise=true , address=145.79.227.11:4729 , piece_id=QYOGBVZV27KVOBIHH3HSFMTZNIU2QDCUKW64QXM2FQRFGU4PRMSA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:45.197419605 (564h1m45.06850265s) [upload=494519]: Next returned nil segment Dec 29 03:30:45.197435445 (564h1m45.06851848s) [upload=494519]: Waiting for error group managing segments... Dec 29 03:30:45.477986829 (564h1m45.349069864s) [noise=true , address=145.79.227.51:4410 , piece_id=N5SEDJHTZZ5VY2TCWKKZXC2TINFLYDIBEEWEZTKZI634XGSI5JOA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.479112532 (564h1m45.350195577s) [noise=true , address=145.79.227.48:4301 , piece_id=GI6HQC3ZYOH7LFFUAAPVKAPTX5R2KIOFWAFP4T46I3GOVVY3TEOQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.485858638 (564h1m45.356941673s) [noise=true , address=145.79.227.13:4803 , piece_id=KW7F7VVT6SUQPJKPHNYYAKKJBTRR4EUABTFFDOWAPW54OIV6HWMQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.507419402 (564h1m45.378502447s) [noise=true , address=145.79.227.50:4375 , piece_id=4XRC5S3ZLO3V3Y45QOGDLDHWQIFW2AOGIUCKJN4Y7YURFSUK4R4A, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.507508672 (564h1m45.378591737s) [noise=true , address=145.79.227.15:4875 , piece_id=CGM4KY3VN4EACJTTL53J234BDEDT7B2JFZPJJQFNKO2XAEMVWEOQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509276008 (564h1m45.380359053s) [noise=true , address=145.79.227.11:4729 , piece_id=QYOGBVZV27KVOBIHH3HSFMTZNIU2QDCUKW64QXM2FQRFGU4PRMSA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509466129 (564h1m45.380549164s) [noise=true , address=145.79.227.7:4293 , piece_id=QF76422GKZBM2TEAEAYLFGQEIJITAHXSWKMQQTJOT4M5MCEACZFA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509699674 (564h1m45.380782709s) [noise=true , address=145.79.227.16:4911 , piece_id=DHHUOKE3J6UIMPXJIDF7XFTJ6AFML4CWGCL7UWFYSTEDLWJK4GWQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509965490 (564h1m45.381048535s) [noise=true , address=145.79.227.5:4223 , piece_id=TRHTPQH7YGRUHPHECPVKFXR2WRHG7HFEBNKBJVBGSQPZYE3WK2JQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.514431544 (564h1m45.385514589s) [noise=true , address=145.79.227.14:4838 , piece_id=2Z6BAR3AM2UPJAUKQ7STBHSCQMXMVJ2XOMHX5W2IA7XZ7GXI4YPA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.528740210 (564h1m45.399823245s) [noise=true , address=145.79.227.47:4265 , piece_id=272MWUCZEUVQFUE3DFQHLAC6GKOU4KHCLWOYCIB6HENN5YFSK2JA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.772021909 (564h1m45.643104944s) [noise=true , address=45.91.116.12:4228 , piece_id=XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.809238664 (564h1m45.680321699s) [noise=true , address=77.90.31.19:4841 , piece_id=MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.809273901 (564h1m45.680356936s) [seg_pos={0 0}, upload=494519]: Segment reached optimal threshold of 13 pieces. Dec 29 03:30:45.809578611 (564h1m45.680661656s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809587999 (564h1m45.680671044s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.809704540 (564h1m45.680787585s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809740489 (564h1m45.680823524s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.809948004 (564h1m45.681031039s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809991167 (564h1m45.681074202s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.810022285 (564h1m45.68110532s) [seg_pos={0 0}, upload=494519]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:30:45.810107047 (564h1m45.681190092s) [upload=494519]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:30:45.810114922 (564h1m45.681197957s) [upload=494519]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:30:46.105682279 (564h1m45.976765324s) [upload=494519]: Done uploading segments... Dec 29 03:30:46.105756730 (564h1m45.976839765s) [upload=494519]: Upload finished. err: <nil> Duration: 30.002933309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 30.000155129s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 30.00018668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................XX.............] duration: 3.910510143s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:31:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.910527125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....X] duration: 30.003523808s trace: https://jaeger.dev.storj.io/trace/15b4410c64f18fce FAIL (2025-12-29 03:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:30:02.538424530 (564h1m2.409507575s) [upload=494511]: Starting upload Dec 29 03:30:02.538542535 (564h1m2.40962558s) [upload=494511]: Done starting upload Dec 29 03:30:02.538764688 (564h1m2.409847733s) [upload=494511]: Uploading segments... Dec 29 03:30:02.538888443 (564h1m2.409971488s) [upload=494511]: Got next segment. Inline: false Dec 29 03:30:02.538941845 (564h1m2.41002489s) [upload=494511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:30:02.538950471 (564h1m2.410033516s) [upload=494511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:30:02.816464684 (564h1m2.687547739s) [seg_pos={0 0}, upload=494511]: Begin upload segment... Dec 29 03:30:02.816528024 (564h1m2.687611069s) [seg_pos={0 0}, upload=494511]: Done begin upload segment. Dec 29 03:30:02.816570826 (564h1m2.687653871s) [noise=true , address=145.79.227.11:4730 , piece_id=PTSDWJSESHAGT7YPGOHQNHLSIKEJIOB5TFSJLNEFLDIG7WQ5BRMA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.816850519 (564h1m2.687933574s) [noise=true , address=145.79.227.19:4077 , piece_id=CPMXMHDMFDT4TF3JN4XK3AQ4P3IEEIKAMWKDKYQFBER4SKNKHTDA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.817418781 (564h1m2.688501836s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.817989828 (564h1m2.689072883s) [noise=true , address=77.90.31.18:4808 , piece_id=E3RVZMHBMHUJ4MSR44RAVMDVTD75KBRKVUFKTMSZKYXOO4CNFT6Q, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.818458651 (564h1m2.689541706s) [noise=true , address=77.90.31.16:4735 , piece_id=426RMS3ER4DPENKIFUXCPQQDKEFG3GDEPRZAXTBKEVBPXQLN2WWA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819168993 (564h1m2.690252048s) [noise=true , address=145.79.227.43:4159 , piece_id=UGEOANOJN333PQUVQY3LXEFN7ZUIRNGCJPZKUMUALIU6QSHWHHNA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819702549 (564h1m2.690785594s) [noise=true , address=145.79.227.47:4267 , piece_id=X54JI7LFNLO4DPA3JRKD7XZDJSUGFQFBIUVQKMVFKURXCFK272YA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819779556 (564h1m2.690862601s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820225995 (564h1m2.69130904s) [noise=true , address=45.91.116.11:4120 , piece_id=IN5ATYACAMGX4MN3HLFR2D4H3P2NJT34EABUHEI4VGL4LWL7DJGA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820242658 (564h1m2.691325703s) [noise=true , address=77.90.31.13:4626 , piece_id=4WG56YDS33OMACKJBQY566DZHA4LRV7NDLDUSP72TZANSF5OXZMQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820726800 (564h1m2.691809845s) [noise=true , address=45.91.116.12:4227 , piece_id=URBYIJUXJGOAZGEZQ6C4IWG57SIOWOG2D7MXG5ZIJSRITG7HNLXA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820745374 (564h1m2.69182842s) [noise=true , address=45.91.116.5:4297 , piece_id=S5IH7YLIGDFSWWPOFS66MLLLM6T5KRC37ATBPTM2PJBDPUKFW5VA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821175664 (564h1m2.692258709s) [noise=true , address=45.91.116.8:4047 , piece_id=IULMJMQL2DMS7JRA53MMWU2AX6KWL4V5UJASQIT7UN64NYGFQ7VQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821200551 (564h1m2.692283596s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821667851 (564h1m2.692750967s) [noise=true , address=45.91.116.9:4263 , piece_id=JCTZTTV4WTYQEXKOOUY6DHZNLUZT4MRU5JUPXTHVSSBOOCOHO4LA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821676788 (564h1m2.692759833s) [noise=true , address=145.79.227.46:4601 , piece_id=XTSXORQ63SD2RJGBDHHX4XDAPFL4XELGWU5WZOOAV3M5EY3UD2BQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:04.457675598 (564h1m4.328758663s) [upload=494511]: Next returned nil segment Dec 29 03:30:04.457693432 (564h1m4.328776477s) [upload=494511]: Waiting for error group managing segments... Dec 29 03:30:04.836499596 (564h1m4.707582641s) [noise=true , address=145.79.227.11:4730 , piece_id=PTSDWJSESHAGT7YPGOHQNHLSIKEJIOB5TFSJLNEFLDIG7WQ5BRMA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.859861289 (564h1m4.730944334s) [noise=true , address=145.79.227.46:4601 , piece_id=XTSXORQ63SD2RJGBDHHX4XDAPFL4XELGWU5WZOOAV3M5EY3UD2BQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.898142420 (564h1m4.769225465s) [noise=true , address=145.79.227.47:4267 , piece_id=X54JI7LFNLO4DPA3JRKD7XZDJSUGFQFBIUVQKMVFKURXCFK272YA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.909411759 (564h1m4.780494804s) [noise=true , address=145.79.227.19:4077 , piece_id=CPMXMHDMFDT4TF3JN4XK3AQ4P3IEEIKAMWKDKYQFBER4SKNKHTDA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.048718477 (564h1m4.919801522s) [noise=true , address=77.90.31.13:4626 , piece_id=4WG56YDS33OMACKJBQY566DZHA4LRV7NDLDUSP72TZANSF5OXZMQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.084691974 (564h1m4.955775019s) [noise=true , address=45.91.116.8:4047 , piece_id=IULMJMQL2DMS7JRA53MMWU2AX6KWL4V5UJASQIT7UN64NYGFQ7VQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.120682234 (564h1m4.991765279s) [noise=true , address=77.90.31.16:4735 , piece_id=426RMS3ER4DPENKIFUXCPQQDKEFG3GDEPRZAXTBKEVBPXQLN2WWA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.140000749 (564h1m5.011083804s) [noise=true , address=77.90.31.18:4808 , piece_id=E3RVZMHBMHUJ4MSR44RAVMDVTD75KBRKVUFKTMSZKYXOO4CNFT6Q, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.187259025 (564h1m5.05834207s) [noise=true , address=45.91.116.5:4297 , piece_id=S5IH7YLIGDFSWWPOFS66MLLLM6T5KRC37ATBPTM2PJBDPUKFW5VA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.207672454 (564h1m5.078755509s) [noise=true , address=45.91.116.11:4120 , piece_id=IN5ATYACAMGX4MN3HLFR2D4H3P2NJT34EABUHEI4VGL4LWL7DJGA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.229462195 (564h1m5.10054527s) [noise=true , address=45.91.116.9:4263 , piece_id=JCTZTTV4WTYQEXKOOUY6DHZNLUZT4MRU5JUPXTHVSSBOOCOHO4LA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.186953075 (564h1m6.05803612s) [noise=true , address=145.79.227.43:4159 , piece_id=UGEOANOJN333PQUVQY3LXEFN7ZUIRNGCJPZKUMUALIU6QSHWHHNA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.554325818 (564h1m6.425408853s) [noise=true , address=45.91.116.12:4227 , piece_id=URBYIJUXJGOAZGEZQ6C4IWG57SIOWOG2D7MXG5ZIJSRITG7HNLXA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.554354872 (564h1m6.425437917s) [seg_pos={0 0}, upload=494511]: Segment reached optimal threshold of 13 pieces. Dec 29 03:30:06.554611562 (564h1m6.425694607s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection Dec 29 03:30:06.554622622 (564h1m6.425705668s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.554939796 (564h1m6.426022851s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:06.554963231 (564h1m6.426046276s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.555230940 (564h1m6.426313985s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Dec 29 03:30:06.555239537 (564h1m6.426322583s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.555257441 (564h1m6.426340486s) [seg_pos={0 0}, upload=494511]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:30:06.555341482 (564h1m6.426424527s) [upload=494511]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:30:06.555347493 (564h1m6.426430538s) [upload=494511]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:30:06.847283826 (564h1m6.718366871s) [upload=494511]: Done uploading segments... Dec 29 03:30:06.847307922 (564h1m6.718390967s) [upload=494511]: Upload finished. err: <nil> Duration: 30.003552623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 4.162507325s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.162535327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................XX...........] duration: 3.673770189s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673840521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 39.117853349s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:30:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.117883466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....] duration: 8.094760579s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 8.094789322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 32.380608726s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 32.380629977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................XX..........] duration: 3.539130749s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539161757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X....] duration: 6.245499181s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:28:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 6.245538026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 3.546031243s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:28:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546116355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................................X.........] duration: 15.412544967s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:28:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 15.412581067s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)