Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X.X.X....X...XXX......X...X....X...X..........X....X.....] duration: 3.639588339s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:09:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.639643794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.] duration: 5.956426308s trace: https://jaeger.dev.storj.io/trace/1d4fe07e577f6bcd FAIL (2025-12-29 11:08:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA) context deadline exceeded ======LOGS====== Dec 29 11:08:09.525720060 (571h39m9.396803095s) [upload=501173]: Starting upload Dec 29 11:08:09.525776498 (571h39m9.396859543s) [upload=501173]: Done starting upload Dec 29 11:08:09.526030010 (571h39m9.397113045s) [upload=501173]: Uploading segments... Dec 29 11:08:09.526085686 (571h39m9.397168721s) [upload=501173]: Got next segment. Inline: false Dec 29 11:08:09.526110132 (571h39m9.397193167s) [upload=501173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:08:09.526114542 (571h39m9.397197577s) [upload=501173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:08:09.772483155 (571h39m9.64356619s) [seg_pos={0 0}, upload=501173]: Begin upload segment... Dec 29 11:08:09.772525816 (571h39m9.643608851s) [seg_pos={0 0}, upload=501173]: Done begin upload segment. Dec 29 11:08:09.772554190 (571h39m9.643637225s) [noise=true , address=45.91.116.9:4263 , piece_id=56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.772562766 (571h39m9.643645801s) [noise=true , address=145.79.227.19:4078 , piece_id=5XJKPI7BM3LMNUP5XDXYV543B6M4XUFPCOIQ7OKCKTXDJWRTL52A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773043121 (571h39m9.644126166s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773286705 (571h39m9.64436974s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773649786 (571h39m9.644732831s) [noise=true , address=45.91.116.7:4010 , piece_id=MDBTZ2NF6I36UIHHFC5UEF5PGAHD2MJTGLFM27MZPCXL6CJVXTGA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774010863 (571h39m9.645093898s) [noise=true , address=145.79.227.39:4015 , piece_id=MXX6ICU5VVPLTPFPZJQNERMT6SHP7XH4FEGWW2WDLCSWR5JUBSDA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774030691 (571h39m9.645113736s) [noise=true , address=145.79.227.6:4257 , piece_id=7WD5TSZT4M7BOT2CGIQOPQLRSD2J453TWRIFHVRVSMCTH7EALTXA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774289874 (571h39m9.645372909s) [noise=true , address=145.79.227.26:4653 , piece_id=YFHBJN4ZFZ6PG3UBOVD2WYTOOFVKXSVVY5QVETXSWF6635EJWHFA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774547064 (571h39m9.645630099s) [noise=true , address=45.91.116.14:4190 , piece_id=SPPSEV2QXWD43BQSUUSY46OBBODCE74O3D7NP7SDAGDG6P2SJ3XQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774847245 (571h39m9.64593029s) [noise=true , address=45.91.116.10:4082 , piece_id=R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775162636 (571h39m9.646245681s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTMY5YXEZVDXNH7ZHFFDFCX5U6WJOJCPDTWF6XXH2GC7PQBSWU3Q, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775439884 (571h39m9.646522919s) [noise=true , address=145.79.227.15:4876 , piece_id=ICMBCMH6SQU2DVU5FY37ZPLF6H3DPGOF55Y2TH3HIPNJCMK2DO3A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775586253 (571h39m9.646669288s) [noise=true , address=45.91.116.13:4155 , piece_id=QZI5RTMLSU3CU63YJKODV2DSYB357CAMEGLSFSRSPBORY4QDFQXQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775978350 (571h39m9.647061385s) [noise=true , address=145.79.227.37:4566 , piece_id=3WOIFGK6KOFWZO5Y7SGT5RSW33YSJID7IW72P3D5M6VF5L4N5RMQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.776393540 (571h39m9.647476585s) [noise=true , address=77.90.31.21:4878 , piece_id=LSV2VMQNX6CBL3UU5QD6622TPHWMQQJZZCAL3EHLYGVVB3WNW6AA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.776502257 (571h39m9.647585302s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:11.476196335 (571h39m11.347279381s) [upload=501173]: Next returned nil segment Dec 29 11:08:11.476221634 (571h39m11.347304679s) [upload=501173]: Waiting for error group managing segments... Dec 29 11:08:11.758907460 (571h39m11.629990505s) [noise=true , address=145.79.227.15:4876 , piece_id=ICMBCMH6SQU2DVU5FY37ZPLF6H3DPGOF55Y2TH3HIPNJCMK2DO3A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.763586549 (571h39m11.634669594s) [noise=true , address=145.79.227.39:4015 , piece_id=MXX6ICU5VVPLTPFPZJQNERMT6SHP7XH4FEGWW2WDLCSWR5JUBSDA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.764007250 (571h39m11.635090295s) [noise=true , address=145.79.227.19:4078 , piece_id=5XJKPI7BM3LMNUP5XDXYV543B6M4XUFPCOIQ7OKCKTXDJWRTL52A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.764589159 (571h39m11.635672194s) [noise=true , address=145.79.227.26:4653 , piece_id=YFHBJN4ZFZ6PG3UBOVD2WYTOOFVKXSVVY5QVETXSWF6635EJWHFA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.779202344 (571h39m11.650285389s) [noise=true , address=145.79.227.6:4257 , piece_id=7WD5TSZT4M7BOT2CGIQOPQLRSD2J453TWRIFHVRVSMCTH7EALTXA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.779766208 (571h39m11.650849253s) [noise=true , address=145.79.227.37:4566 , piece_id=3WOIFGK6KOFWZO5Y7SGT5RSW33YSJID7IW72P3D5M6VF5L4N5RMQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.958594706 (571h39m11.829677751s) [noise=true , address=77.90.31.21:4878 , piece_id=LSV2VMQNX6CBL3UU5QD6622TPHWMQQJZZCAL3EHLYGVVB3WNW6AA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.979826504 (571h39m11.850909539s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTMY5YXEZVDXNH7ZHFFDFCX5U6WJOJCPDTWF6XXH2GC7PQBSWU3Q, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.026421537 (571h39m11.897504572s) [noise=true , address=45.91.116.14:4190 , piece_id=SPPSEV2QXWD43BQSUUSY46OBBODCE74O3D7NP7SDAGDG6P2SJ3XQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.030432345 (571h39m11.90151539s) [noise=true , address=45.91.116.9:4263 , piece_id=56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.058341237 (571h39m11.929424282s) [noise=true , address=45.91.116.10:4082 , piece_id=R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.068152479 (571h39m11.939235544s) [noise=true , address=45.91.116.13:4155 , piece_id=QZI5RTMLSU3CU63YJKODV2DSYB357CAMEGLSFSRSPBORY4QDFQXQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.137712102 (571h39m12.008795137s) [noise=true , address=45.91.116.7:4010 , piece_id=MDBTZ2NF6I36UIHHFC5UEF5PGAHD2MJTGLFM27MZPCXL6CJVXTGA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.137731879 (571h39m12.008814914s) [seg_pos={0 0}, upload=501173]: Segment reached optimal threshold of 13 pieces. Dec 29 11:08:12.137952810 (571h39m12.009035855s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 11:08:12.137959823 (571h39m12.009042858s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138143332 (571h39m12.009226367s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 11:08:12.138150165 (571h39m12.0092332s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138735340 (571h39m12.009818385s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection Dec 29 11:08:12.138767932 (571h39m12.009850977s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138795895 (571h39m12.00987904s) [seg_pos={0 0}, upload=501173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:08:12.138900975 (571h39m12.00998402s) [upload=501173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:08:12.138907718 (571h39m12.009990763s) [upload=501173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:08:12.434939406 (571h39m12.306022451s) [upload=501173]: Done uploading segments... Dec 29 11:08:12.434966598 (571h39m12.306049643s) [upload=501173]: Upload finished. err: <nil> Duration: 5.956444513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X...X.X] duration: 30.002812577s trace: https://jaeger.dev.storj.io/trace/1d4fe07e577f6bcd FAIL (2025-12-29 11:08:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA) context deadline exceeded ======LOGS====== Dec 29 11:08:09.525720060 (571h39m9.396803095s) [upload=501173]: Starting upload Dec 29 11:08:09.525776498 (571h39m9.396859543s) [upload=501173]: Done starting upload Dec 29 11:08:09.526030010 (571h39m9.397113045s) [upload=501173]: Uploading segments... Dec 29 11:08:09.526085686 (571h39m9.397168721s) [upload=501173]: Got next segment. Inline: false Dec 29 11:08:09.526110132 (571h39m9.397193167s) [upload=501173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:08:09.526114542 (571h39m9.397197577s) [upload=501173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:08:09.772483155 (571h39m9.64356619s) [seg_pos={0 0}, upload=501173]: Begin upload segment... Dec 29 11:08:09.772525816 (571h39m9.643608851s) [seg_pos={0 0}, upload=501173]: Done begin upload segment. Dec 29 11:08:09.772554190 (571h39m9.643637225s) [noise=true , address=45.91.116.9:4263 , piece_id=56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.772562766 (571h39m9.643645801s) [noise=true , address=145.79.227.19:4078 , piece_id=5XJKPI7BM3LMNUP5XDXYV543B6M4XUFPCOIQ7OKCKTXDJWRTL52A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773043121 (571h39m9.644126166s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773286705 (571h39m9.64436974s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.773649786 (571h39m9.644732831s) [noise=true , address=45.91.116.7:4010 , piece_id=MDBTZ2NF6I36UIHHFC5UEF5PGAHD2MJTGLFM27MZPCXL6CJVXTGA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774010863 (571h39m9.645093898s) [noise=true , address=145.79.227.39:4015 , piece_id=MXX6ICU5VVPLTPFPZJQNERMT6SHP7XH4FEGWW2WDLCSWR5JUBSDA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774030691 (571h39m9.645113736s) [noise=true , address=145.79.227.6:4257 , piece_id=7WD5TSZT4M7BOT2CGIQOPQLRSD2J453TWRIFHVRVSMCTH7EALTXA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774289874 (571h39m9.645372909s) [noise=true , address=145.79.227.26:4653 , piece_id=YFHBJN4ZFZ6PG3UBOVD2WYTOOFVKXSVVY5QVETXSWF6635EJWHFA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774547064 (571h39m9.645630099s) [noise=true , address=45.91.116.14:4190 , piece_id=SPPSEV2QXWD43BQSUUSY46OBBODCE74O3D7NP7SDAGDG6P2SJ3XQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.774847245 (571h39m9.64593029s) [noise=true , address=45.91.116.10:4082 , piece_id=R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775162636 (571h39m9.646245681s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTMY5YXEZVDXNH7ZHFFDFCX5U6WJOJCPDTWF6XXH2GC7PQBSWU3Q, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775439884 (571h39m9.646522919s) [noise=true , address=145.79.227.15:4876 , piece_id=ICMBCMH6SQU2DVU5FY37ZPLF6H3DPGOF55Y2TH3HIPNJCMK2DO3A, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775586253 (571h39m9.646669288s) [noise=true , address=45.91.116.13:4155 , piece_id=QZI5RTMLSU3CU63YJKODV2DSYB357CAMEGLSFSRSPBORY4QDFQXQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.775978350 (571h39m9.647061385s) [noise=true , address=145.79.227.37:4566 , piece_id=3WOIFGK6KOFWZO5Y7SGT5RSW33YSJID7IW72P3D5M6VF5L4N5RMQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.776393540 (571h39m9.647476585s) [noise=true , address=77.90.31.21:4878 , piece_id=LSV2VMQNX6CBL3UU5QD6622TPHWMQQJZZCAL3EHLYGVVB3WNW6AA, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:09.776502257 (571h39m9.647585302s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Uploading piece... Dec 29 11:08:11.476196335 (571h39m11.347279381s) [upload=501173]: Next returned nil segment Dec 29 11:08:11.476221634 (571h39m11.347304679s) [upload=501173]: Waiting for error group managing segments... Dec 29 11:08:11.758907460 (571h39m11.629990505s) [noise=true , address=145.79.227.15:4876 , piece_id=ICMBCMH6SQU2DVU5FY37ZPLF6H3DPGOF55Y2TH3HIPNJCMK2DO3A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.763586549 (571h39m11.634669594s) [noise=true , address=145.79.227.39:4015 , piece_id=MXX6ICU5VVPLTPFPZJQNERMT6SHP7XH4FEGWW2WDLCSWR5JUBSDA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.764007250 (571h39m11.635090295s) [noise=true , address=145.79.227.19:4078 , piece_id=5XJKPI7BM3LMNUP5XDXYV543B6M4XUFPCOIQ7OKCKTXDJWRTL52A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.764589159 (571h39m11.635672194s) [noise=true , address=145.79.227.26:4653 , piece_id=YFHBJN4ZFZ6PG3UBOVD2WYTOOFVKXSVVY5QVETXSWF6635EJWHFA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.779202344 (571h39m11.650285389s) [noise=true , address=145.79.227.6:4257 , piece_id=7WD5TSZT4M7BOT2CGIQOPQLRSD2J453TWRIFHVRVSMCTH7EALTXA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.779766208 (571h39m11.650849253s) [noise=true , address=145.79.227.37:4566 , piece_id=3WOIFGK6KOFWZO5Y7SGT5RSW33YSJID7IW72P3D5M6VF5L4N5RMQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.958594706 (571h39m11.829677751s) [noise=true , address=77.90.31.21:4878 , piece_id=LSV2VMQNX6CBL3UU5QD6622TPHWMQQJZZCAL3EHLYGVVB3WNW6AA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:11.979826504 (571h39m11.850909539s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTMY5YXEZVDXNH7ZHFFDFCX5U6WJOJCPDTWF6XXH2GC7PQBSWU3Q, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.026421537 (571h39m11.897504572s) [noise=true , address=45.91.116.14:4190 , piece_id=SPPSEV2QXWD43BQSUUSY46OBBODCE74O3D7NP7SDAGDG6P2SJ3XQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.030432345 (571h39m11.90151539s) [noise=true , address=45.91.116.9:4263 , piece_id=56QRB44GXI3ORHVL6RNGMI5WESNCP52Z3CGX6TIYVQQQMWYHAS3A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.058341237 (571h39m11.929424282s) [noise=true , address=45.91.116.10:4082 , piece_id=R5XKOTPQKY6HOPHGJAF7DVZ4TUSMAIHNAQGAMRBPD5343IYFZ3JA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.068152479 (571h39m11.939235544s) [noise=true , address=45.91.116.13:4155 , piece_id=QZI5RTMLSU3CU63YJKODV2DSYB357CAMEGLSFSRSPBORY4QDFQXQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.137712102 (571h39m12.008795137s) [noise=true , address=45.91.116.7:4010 , piece_id=MDBTZ2NF6I36UIHHFC5UEF5PGAHD2MJTGLFM27MZPCXL6CJVXTGA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: <nil> Dec 29 11:08:12.137731879 (571h39m12.008814914s) [seg_pos={0 0}, upload=501173]: Segment reached optimal threshold of 13 pieces. Dec 29 11:08:12.137952810 (571h39m12.009035855s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 11:08:12.137959823 (571h39m12.009042858s) [noise=true , address=145.79.227.40:4049 , piece_id=PULKCLLJBP3KLXJC54HDVUXYT2LQ7RLUOFRNOZNFOKUVEH2ZAUJA, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138143332 (571h39m12.009226367s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 11:08:12.138150165 (571h39m12.0092332s) [noise=true , address=45.91.116.6:4974 , piece_id=ULRIYXIUMNGETEMRKMPX6EH4SC6ORZ5CWDRSWMSD42JHBYRLWPWQ, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138735340 (571h39m12.009818385s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->77.90.31.18:4805: use of closed network connection Dec 29 11:08:12.138767932 (571h39m12.009850977s) [noise=true , address=77.90.31.18:4805 , piece_id=5O54PLV3CI74PN4HPZGMUMZZVSVBLU5CUIJDRKML3ILZRCPZZX5A, seg_pos={0 0}, upload=501173]: Long tail cancellation due to optimal threshold Dec 29 11:08:12.138795895 (571h39m12.00987904s) [seg_pos={0 0}, upload=501173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:08:12.138900975 (571h39m12.00998402s) [upload=501173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:08:12.138907718 (571h39m12.009990763s) [upload=501173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:08:12.434939406 (571h39m12.306022451s) [upload=501173]: Done uploading segments... Dec 29 11:08:12.434966598 (571h39m12.306049643s) [upload=501173]: Upload finished. err: <nil> Duration: 30.002834298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..................................X...] duration: 15.361130971s trace: https://jaeger.dev.storj.io/trace/2a822012276bcc5d FAIL (2025-12-29 11:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882470730 (571h36m32.753553775s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882771553 (571h36m32.753854588s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883107963 (571h36m32.754190998s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883533073 (571h36m32.754616118s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883944547 (571h36m32.755027582s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.884326042 (571h36m32.755409077s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886067960 (571h36m32.757150995s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886589914 (571h36m32.757672949s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886895325 (571h36m32.75797837s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888604990 (571h36m32.759688035s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888877790 (571h36m32.759960925s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.890542230 (571h36m32.761625275s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.893184912 (571h36m32.764267947s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.896004932 (571h36m32.767087967s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.901393461 (571h36m32.772476507s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:34.200413331 (571h36m34.071496366s) [upload=501136]: Next returned nil segment Dec 29 11:05:34.200437376 (571h36m34.071520411s) [upload=501136]: Waiting for error group managing segments... Dec 29 11:05:34.531472608 (571h36m34.402555643s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.589291783 (571h36m34.460374819s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.782400973 (571h36m34.653484008s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.895054052 (571h36m34.766137097s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.965988150 (571h36m34.837071195s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.068876977 (571h36m34.939960012s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.075857198 (571h36m34.946940243s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.237630089 (571h36m35.108713134s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.371788215 (571h36m35.24287126s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.635729582 (571h36m35.506812627s) [noise=true , address=145.79.227.30:4784 , piece_id=BUNTKFESZUDAYXSOIMKUG7DVUF65CQJAHDB7NVHII4A5LEWA4K4Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.650702903 (571h36m35.521785948s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.721297946 (571h36m35.592380991s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.812319244 (571h36m35.683402279s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101131227 (571h36m35.972214262s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101172385 (571h36m35.97225542s) [seg_pos={0 0}, upload=501136]: Segment reached optimal threshold of 14 pieces. Dec 29 11:05:36.101312391 (571h36m35.972395436s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101318363 (571h36m35.972401408s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101621961 (571h36m35.972704996s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection Dec 29 11:05:36.101629084 (571h36m35.972712119s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101715780 (571h36m35.972798815s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DdwuGVzn339aKBRre1NvC9rUwSbL8v4GLwWj5TZBRP6RvLDnF): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101721320 (571h36m35.972804355s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101848852 (571h36m35.972931887s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101854173 (571h36m35.972937208s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101868661 (571h36m35.972951706s) [seg_pos={0 0}, upload=501136]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:05:36.101932713 (571h36m35.973015748s) [upload=501136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:05:36.101937271 (571h36m35.973020316s) [upload=501136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:05:36.403553304 (571h36m36.274636349s) [upload=501136]: Done uploading segments... Dec 29 11:05:36.403573943 (571h36m36.274656978s) [upload=501136]: Upload finished. err: <nil> Duration: 15.36116202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X...X.X.X....X...XXX......X...X....X...X..........X....X....] duration: 3.568090757s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:07:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.568119713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X...X.] duration: 22.435077716s trace: https://jaeger.dev.storj.io/trace/3cd531538558c698 FAIL (2025-12-29 11:07:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:07:00.396567008 (571h38m0.267650053s) [upload=501156]: Starting upload Dec 29 11:07:00.396642812 (571h38m0.267725867s) [upload=501156]: Done starting upload Dec 29 11:07:00.396964415 (571h38m0.26804746s) [upload=501156]: Uploading segments... Dec 29 11:07:00.397059656 (571h38m0.268142701s) [upload=501156]: Got next segment. Inline: false Dec 29 11:07:00.397117706 (571h38m0.268200741s) [upload=501156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:07:00.397124048 (571h38m0.268207083s) [upload=501156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:07:00.645768505 (571h38m0.51685155s) [seg_pos={0 0}, upload=501156]: Begin upload segment... Dec 29 11:07:00.645851062 (571h38m0.516934097s) [seg_pos={0 0}, upload=501156]: Done begin upload segment. Dec 29 11:07:00.645896418 (571h38m0.516979453s) [noise=true , address=145.79.227.45:4230 , piece_id=D225Q23JOZJ4TOM37IC4WFT6IAJP557H42CE5QJCZR7WQYUOFHWQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646206068 (571h38m0.517289113s) [noise=true , address=145.79.227.42:4123 , piece_id=S4ZFTIVYKZM337WXDCRCEY3VQ3J55YJVLNYTM42Z7S3I2NEJALTA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646578658 (571h38m0.517661703s) [noise=true , address=145.79.227.18:4007 , piece_id=EVIO5EXSSBEFU6KYQXYNYGPY4OCGFRFBEXCRZD65NB6VZ7V3TWQQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646870113 (571h38m0.517953148s) [noise=true , address=145.79.227.39:4013 , piece_id=T3EMIJ5QCL2GQ5BCT5M56PRCGIPKBIEXLQL7OYT3HRKWDPXCNYPA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646992496 (571h38m0.518075531s) [noise=true , address=145.79.227.21:4473 , piece_id=ZYVLAQQYL6NMQQBSAS25JUPCLYJUIFR233BPN2PYZRFMEUNBVJKQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647282298 (571h38m0.518365343s) [noise=true , address=145.79.227.14:4839 , piece_id=JPUNE4QLHMZOBR6QW35KJIERU44JT3N72JCD3TZHGC332HI36WPA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647410793 (571h38m0.518493828s) [noise=true , address=145.79.227.38:4977 , piece_id=EA2VSGSHIMZRFQCK3PVCBC37AOF2HGBZ3AVTZ5OWMM4N2YTTIHLQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647629709 (571h38m0.518712754s) [noise=true , address=45.91.116.5:4298 , piece_id=OUO7QWLUAZLBZB3RPVOYUI2LYXNUWU7ZR7CAL7ZPILGKCLCNEQVQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647896528 (571h38m0.518979573s) [noise=true , address=45.91.116.14:4192 , piece_id=OKZCRXKBRXOC53UIGLL3F3HHBNHDTHS75LVNXEDEW3QX3GS2J4VA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647937575 (571h38m0.519020621s) [noise=true , address=77.90.31.14:4661 , piece_id=GBQPY7YY6HRJQT6IIJITTZILQXBHBU74ALGNZCFZV5BEYQZAAK3A, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648430163 (571h38m0.519513208s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648468035 (571h38m0.51955109s) [noise=true , address=145.79.227.3:4151 , piece_id=A3HKBWG4BB3AK6IM4T5AZDIKB5Z45AKMZ2ODKXVHCXIZVLYL7IYQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648861264 (571h38m0.519944309s) [noise=true , address=145.79.227.29:4942 , piece_id=LMWDV55Y2JSMHWFNEOGOG3GTZWZWPXGJC4BRXWABU4RNRHFQTTBA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648901971 (571h38m0.519985016s) [noise=true , address=145.79.227.12:4767 , piece_id=4GSRXI7WY75Z33PBNJDYG4MAXAJOSJB6MM4ZAJSDBK7NXRJDOW3A, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.649257840 (571h38m0.520340885s) [noise=true , address=45.91.116.7:4011 , piece_id=CGASR77FC6FKSMGG4KWMAXXPKHYDG6RO7CXACKVGO6TNBNRA2DRQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.649466787 (571h38m0.520549823s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:02.255867068 (571h38m2.126950113s) [upload=501156]: Next returned nil segment Dec 29 11:07:02.255884521 (571h38m2.126967556s) [upload=501156]: Waiting for error group managing segments... Dec 29 11:07:02.639922629 (571h38m2.511005674s) [noise=true , address=145.79.227.12:4767 , piece_id=4GSRXI7WY75Z33PBNJDYG4MAXAJOSJB6MM4ZAJSDBK7NXRJDOW3A, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.640076822 (571h38m2.511159857s) [noise=true , address=145.79.227.14:4839 , piece_id=JPUNE4QLHMZOBR6QW35KJIERU44JT3N72JCD3TZHGC332HI36WPA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.641445668 (571h38m2.512528753s) [noise=true , address=145.79.227.3:4151 , piece_id=A3HKBWG4BB3AK6IM4T5AZDIKB5Z45AKMZ2ODKXVHCXIZVLYL7IYQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.642053035 (571h38m2.51313608s) [noise=true , address=145.79.227.39:4013 , piece_id=T3EMIJ5QCL2GQ5BCT5M56PRCGIPKBIEXLQL7OYT3HRKWDPXCNYPA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.643037990 (571h38m2.514121035s) [noise=true , address=145.79.227.42:4123 , piece_id=S4ZFTIVYKZM337WXDCRCEY3VQ3J55YJVLNYTM42Z7S3I2NEJALTA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.643756840 (571h38m2.514839885s) [noise=true , address=145.79.227.38:4977 , piece_id=EA2VSGSHIMZRFQCK3PVCBC37AOF2HGBZ3AVTZ5OWMM4N2YTTIHLQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.648813749 (571h38m2.519896794s) [noise=true , address=145.79.227.29:4942 , piece_id=LMWDV55Y2JSMHWFNEOGOG3GTZWZWPXGJC4BRXWABU4RNRHFQTTBA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.652134612 (571h38m2.523217657s) [noise=true , address=145.79.227.21:4473 , piece_id=ZYVLAQQYL6NMQQBSAS25JUPCLYJUIFR233BPN2PYZRFMEUNBVJKQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.661665631 (571h38m2.532748676s) [noise=true , address=145.79.227.45:4230 , piece_id=D225Q23JOZJ4TOM37IC4WFT6IAJP557H42CE5QJCZR7WQYUOFHWQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.663961422 (571h38m2.535044467s) [noise=true , address=145.79.227.18:4007 , piece_id=EVIO5EXSSBEFU6KYQXYNYGPY4OCGFRFBEXCRZD65NB6VZ7V3TWQQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.886535717 (571h38m2.757618762s) [noise=true , address=45.91.116.5:4298 , piece_id=OUO7QWLUAZLBZB3RPVOYUI2LYXNUWU7ZR7CAL7ZPILGKCLCNEQVQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.925762681 (571h38m2.796845726s) [noise=true , address=45.91.116.7:4011 , piece_id=CGASR77FC6FKSMGG4KWMAXXPKHYDG6RO7CXACKVGO6TNBNRA2DRQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.939597415 (571h38m2.81068046s) [noise=true , address=77.90.31.14:4661 , piece_id=GBQPY7YY6HRJQT6IIJITTZILQXBHBU74ALGNZCFZV5BEYQZAAK3A, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.939619036 (571h38m2.810702081s) [seg_pos={0 0}, upload=501156]: Segment reached optimal threshold of 13 pieces. Dec 29 11:07:02.940047532 (571h38m2.811130567s) [noise=true , address=45.91.116.14:4192 , piece_id=OKZCRXKBRXOC53UIGLL3F3HHBNHDTHS75LVNXEDEW3QX3GS2J4VA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.940320862 (571h38m2.811403917s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 11:07:02.940332354 (571h38m2.811415419s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Long tail cancellation due to optimal threshold Dec 29 11:07:02.940523879 (571h38m2.811606914s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 11:07:02.940533577 (571h38m2.811616612s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Long tail cancellation due to optimal threshold Dec 29 11:07:02.940557542 (571h38m2.811640587s) [seg_pos={0 0}, upload=501156]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 11:07:02.940688983 (571h38m2.811772018s) [upload=501156]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:07:02.940694543 (571h38m2.811777588s) [upload=501156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:07:03.238310891 (571h38m3.109393936s) [upload=501156]: Done uploading segments... Dec 29 11:07:03.238342591 (571h38m3.109425636s) [upload=501156]: Upload finished. err: <nil> Duration: 22.43509528s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X...X.X.X....X...XXX......X...X....X...X..........X....X...] duration: 4.080480035s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:07:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.08050376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X...X] duration: 30.002184108s trace: https://jaeger.dev.storj.io/trace/3cd531538558c698 FAIL (2025-12-29 11:07:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:07:00.396567008 (571h38m0.267650053s) [upload=501156]: Starting upload Dec 29 11:07:00.396642812 (571h38m0.267725867s) [upload=501156]: Done starting upload Dec 29 11:07:00.396964415 (571h38m0.26804746s) [upload=501156]: Uploading segments... Dec 29 11:07:00.397059656 (571h38m0.268142701s) [upload=501156]: Got next segment. Inline: false Dec 29 11:07:00.397117706 (571h38m0.268200741s) [upload=501156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:07:00.397124048 (571h38m0.268207083s) [upload=501156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:07:00.645768505 (571h38m0.51685155s) [seg_pos={0 0}, upload=501156]: Begin upload segment... Dec 29 11:07:00.645851062 (571h38m0.516934097s) [seg_pos={0 0}, upload=501156]: Done begin upload segment. Dec 29 11:07:00.645896418 (571h38m0.516979453s) [noise=true , address=145.79.227.45:4230 , piece_id=D225Q23JOZJ4TOM37IC4WFT6IAJP557H42CE5QJCZR7WQYUOFHWQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646206068 (571h38m0.517289113s) [noise=true , address=145.79.227.42:4123 , piece_id=S4ZFTIVYKZM337WXDCRCEY3VQ3J55YJVLNYTM42Z7S3I2NEJALTA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646578658 (571h38m0.517661703s) [noise=true , address=145.79.227.18:4007 , piece_id=EVIO5EXSSBEFU6KYQXYNYGPY4OCGFRFBEXCRZD65NB6VZ7V3TWQQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646870113 (571h38m0.517953148s) [noise=true , address=145.79.227.39:4013 , piece_id=T3EMIJ5QCL2GQ5BCT5M56PRCGIPKBIEXLQL7OYT3HRKWDPXCNYPA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.646992496 (571h38m0.518075531s) [noise=true , address=145.79.227.21:4473 , piece_id=ZYVLAQQYL6NMQQBSAS25JUPCLYJUIFR233BPN2PYZRFMEUNBVJKQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647282298 (571h38m0.518365343s) [noise=true , address=145.79.227.14:4839 , piece_id=JPUNE4QLHMZOBR6QW35KJIERU44JT3N72JCD3TZHGC332HI36WPA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647410793 (571h38m0.518493828s) [noise=true , address=145.79.227.38:4977 , piece_id=EA2VSGSHIMZRFQCK3PVCBC37AOF2HGBZ3AVTZ5OWMM4N2YTTIHLQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647629709 (571h38m0.518712754s) [noise=true , address=45.91.116.5:4298 , piece_id=OUO7QWLUAZLBZB3RPVOYUI2LYXNUWU7ZR7CAL7ZPILGKCLCNEQVQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647896528 (571h38m0.518979573s) [noise=true , address=45.91.116.14:4192 , piece_id=OKZCRXKBRXOC53UIGLL3F3HHBNHDTHS75LVNXEDEW3QX3GS2J4VA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.647937575 (571h38m0.519020621s) [noise=true , address=77.90.31.14:4661 , piece_id=GBQPY7YY6HRJQT6IIJITTZILQXBHBU74ALGNZCFZV5BEYQZAAK3A, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648430163 (571h38m0.519513208s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648468035 (571h38m0.51955109s) [noise=true , address=145.79.227.3:4151 , piece_id=A3HKBWG4BB3AK6IM4T5AZDIKB5Z45AKMZ2ODKXVHCXIZVLYL7IYQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648861264 (571h38m0.519944309s) [noise=true , address=145.79.227.29:4942 , piece_id=LMWDV55Y2JSMHWFNEOGOG3GTZWZWPXGJC4BRXWABU4RNRHFQTTBA, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.648901971 (571h38m0.519985016s) [noise=true , address=145.79.227.12:4767 , piece_id=4GSRXI7WY75Z33PBNJDYG4MAXAJOSJB6MM4ZAJSDBK7NXRJDOW3A, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.649257840 (571h38m0.520340885s) [noise=true , address=45.91.116.7:4011 , piece_id=CGASR77FC6FKSMGG4KWMAXXPKHYDG6RO7CXACKVGO6TNBNRA2DRQ, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:00.649466787 (571h38m0.520549823s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Uploading piece... Dec 29 11:07:02.255867068 (571h38m2.126950113s) [upload=501156]: Next returned nil segment Dec 29 11:07:02.255884521 (571h38m2.126967556s) [upload=501156]: Waiting for error group managing segments... Dec 29 11:07:02.639922629 (571h38m2.511005674s) [noise=true , address=145.79.227.12:4767 , piece_id=4GSRXI7WY75Z33PBNJDYG4MAXAJOSJB6MM4ZAJSDBK7NXRJDOW3A, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.640076822 (571h38m2.511159857s) [noise=true , address=145.79.227.14:4839 , piece_id=JPUNE4QLHMZOBR6QW35KJIERU44JT3N72JCD3TZHGC332HI36WPA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.641445668 (571h38m2.512528753s) [noise=true , address=145.79.227.3:4151 , piece_id=A3HKBWG4BB3AK6IM4T5AZDIKB5Z45AKMZ2ODKXVHCXIZVLYL7IYQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.642053035 (571h38m2.51313608s) [noise=true , address=145.79.227.39:4013 , piece_id=T3EMIJ5QCL2GQ5BCT5M56PRCGIPKBIEXLQL7OYT3HRKWDPXCNYPA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.643037990 (571h38m2.514121035s) [noise=true , address=145.79.227.42:4123 , piece_id=S4ZFTIVYKZM337WXDCRCEY3VQ3J55YJVLNYTM42Z7S3I2NEJALTA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.643756840 (571h38m2.514839885s) [noise=true , address=145.79.227.38:4977 , piece_id=EA2VSGSHIMZRFQCK3PVCBC37AOF2HGBZ3AVTZ5OWMM4N2YTTIHLQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.648813749 (571h38m2.519896794s) [noise=true , address=145.79.227.29:4942 , piece_id=LMWDV55Y2JSMHWFNEOGOG3GTZWZWPXGJC4BRXWABU4RNRHFQTTBA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.652134612 (571h38m2.523217657s) [noise=true , address=145.79.227.21:4473 , piece_id=ZYVLAQQYL6NMQQBSAS25JUPCLYJUIFR233BPN2PYZRFMEUNBVJKQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.661665631 (571h38m2.532748676s) [noise=true , address=145.79.227.45:4230 , piece_id=D225Q23JOZJ4TOM37IC4WFT6IAJP557H42CE5QJCZR7WQYUOFHWQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.663961422 (571h38m2.535044467s) [noise=true , address=145.79.227.18:4007 , piece_id=EVIO5EXSSBEFU6KYQXYNYGPY4OCGFRFBEXCRZD65NB6VZ7V3TWQQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.886535717 (571h38m2.757618762s) [noise=true , address=45.91.116.5:4298 , piece_id=OUO7QWLUAZLBZB3RPVOYUI2LYXNUWU7ZR7CAL7ZPILGKCLCNEQVQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.925762681 (571h38m2.796845726s) [noise=true , address=45.91.116.7:4011 , piece_id=CGASR77FC6FKSMGG4KWMAXXPKHYDG6RO7CXACKVGO6TNBNRA2DRQ, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.939597415 (571h38m2.81068046s) [noise=true , address=77.90.31.14:4661 , piece_id=GBQPY7YY6HRJQT6IIJITTZILQXBHBU74ALGNZCFZV5BEYQZAAK3A, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.939619036 (571h38m2.810702081s) [seg_pos={0 0}, upload=501156]: Segment reached optimal threshold of 13 pieces. Dec 29 11:07:02.940047532 (571h38m2.811130567s) [noise=true , address=45.91.116.14:4192 , piece_id=OKZCRXKBRXOC53UIGLL3F3HHBNHDTHS75LVNXEDEW3QX3GS2J4VA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: <nil> Dec 29 11:07:02.940320862 (571h38m2.811403917s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 11:07:02.940332354 (571h38m2.811415419s) [noise=true , address=77.90.31.13:4626 , piece_id=EW4IWSWML5CEENIS3SEKXIQKAW3HC66ECC2COF4DXPHNUQOYIIOA, seg_pos={0 0}, upload=501156]: Long tail cancellation due to optimal threshold Dec 29 11:07:02.940523879 (571h38m2.811606914s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 11:07:02.940533577 (571h38m2.811616612s) [noise=true , address=77.90.31.12:4590 , piece_id=PQOBI4DPMQBKODJOD75F5UOSAEIK7CMVXSPAU64ZX47XTXSCY35Q, seg_pos={0 0}, upload=501156]: Long tail cancellation due to optimal threshold Dec 29 11:07:02.940557542 (571h38m2.811640587s) [seg_pos={0 0}, upload=501156]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 11:07:02.940688983 (571h38m2.811772018s) [upload=501156]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:07:02.940694543 (571h38m2.811777588s) [upload=501156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:07:03.238310891 (571h38m3.109393936s) [upload=501156]: Done uploading segments... Dec 29 11:07:03.238342591 (571h38m3.109425636s) [upload=501156]: Upload finished. err: <nil> Duration: 30.002204727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..................................X..] duration: 15.307617442s trace: https://jaeger.dev.storj.io/trace/2a822012276bcc5d FAIL (2025-12-29 11:07:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882470730 (571h36m32.753553775s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882771553 (571h36m32.753854588s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883107963 (571h36m32.754190998s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883533073 (571h36m32.754616118s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883944547 (571h36m32.755027582s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.884326042 (571h36m32.755409077s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886067960 (571h36m32.757150995s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886589914 (571h36m32.757672949s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886895325 (571h36m32.75797837s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888604990 (571h36m32.759688035s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888877790 (571h36m32.759960925s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.890542230 (571h36m32.761625275s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.893184912 (571h36m32.764267947s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.896004932 (571h36m32.767087967s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.901393461 (571h36m32.772476507s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:34.200413331 (571h36m34.071496366s) [upload=501136]: Next returned nil segment Dec 29 11:05:34.200437376 (571h36m34.071520411s) [upload=501136]: Waiting for error group managing segments... Dec 29 11:05:34.531472608 (571h36m34.402555643s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.589291783 (571h36m34.460374819s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.782400973 (571h36m34.653484008s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.895054052 (571h36m34.766137097s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.965988150 (571h36m34.837071195s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.068876977 (571h36m34.939960012s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.075857198 (571h36m34.946940243s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.237630089 (571h36m35.108713134s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.371788215 (571h36m35.24287126s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.635729582 (571h36m35.506812627s) [noise=true , address=145.79.227.30:4784 , piece_id=BUNTKFESZUDAYXSOIMKUG7DVUF65CQJAHDB7NVHII4A5LEWA4K4Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.650702903 (571h36m35.521785948s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.721297946 (571h36m35.592380991s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.812319244 (571h36m35.683402279s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101131227 (571h36m35.972214262s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101172385 (571h36m35.97225542s) [seg_pos={0 0}, upload=501136]: Segment reached optimal threshold of 14 pieces. Dec 29 11:05:36.101312391 (571h36m35.972395436s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101318363 (571h36m35.972401408s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101621961 (571h36m35.972704996s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection Dec 29 11:05:36.101629084 (571h36m35.972712119s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101715780 (571h36m35.972798815s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DdwuGVzn339aKBRre1NvC9rUwSbL8v4GLwWj5TZBRP6RvLDnF): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101721320 (571h36m35.972804355s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101848852 (571h36m35.972931887s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101854173 (571h36m35.972937208s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101868661 (571h36m35.972951706s) [seg_pos={0 0}, upload=501136]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:05:36.101932713 (571h36m35.973015748s) [upload=501136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:05:36.101937271 (571h36m35.973020316s) [upload=501136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:05:36.403553304 (571h36m36.274636349s) [upload=501136]: Done uploading segments... Dec 29 11:05:36.403573943 (571h36m36.274656978s) [upload=501136]: Upload finished. err: <nil> Duration: 15.307633823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X...X.X.X....X...XXX......X...X....X...X..........X....X..] duration: 3.493288764s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:07:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493314081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..................................X.] duration: 35.866306527s trace: https://jaeger.dev.storj.io/trace/2a822012276bcc5d FAIL (2025-12-29 11:07:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882470730 (571h36m32.753553775s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882771553 (571h36m32.753854588s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883107963 (571h36m32.754190998s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883533073 (571h36m32.754616118s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883944547 (571h36m32.755027582s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.884326042 (571h36m32.755409077s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886067960 (571h36m32.757150995s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886589914 (571h36m32.757672949s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886895325 (571h36m32.75797837s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888604990 (571h36m32.759688035s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888877790 (571h36m32.759960925s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.890542230 (571h36m32.761625275s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.893184912 (571h36m32.764267947s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.896004932 (571h36m32.767087967s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.901393461 (571h36m32.772476507s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:34.200413331 (571h36m34.071496366s) [upload=501136]: Next returned nil segment Dec 29 11:05:34.200437376 (571h36m34.071520411s) [upload=501136]: Waiting for error group managing segments... Dec 29 11:05:34.531472608 (571h36m34.402555643s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.589291783 (571h36m34.460374819s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.782400973 (571h36m34.653484008s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.895054052 (571h36m34.766137097s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.965988150 (571h36m34.837071195s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.068876977 (571h36m34.939960012s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.075857198 (571h36m34.946940243s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.237630089 (571h36m35.108713134s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.371788215 (571h36m35.24287126s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.635729582 (571h36m35.506812627s) [noise=true , address=145.79.227.30:4784 , piece_id=BUNTKFESZUDAYXSOIMKUG7DVUF65CQJAHDB7NVHII4A5LEWA4K4Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.650702903 (571h36m35.521785948s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.721297946 (571h36m35.592380991s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.812319244 (571h36m35.683402279s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101131227 (571h36m35.972214262s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101172385 (571h36m35.97225542s) [seg_pos={0 0}, upload=501136]: Segment reached optimal threshold of 14 pieces. Dec 29 11:05:36.101312391 (571h36m35.972395436s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101318363 (571h36m35.972401408s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101621961 (571h36m35.972704996s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection Dec 29 11:05:36.101629084 (571h36m35.972712119s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101715780 (571h36m35.972798815s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DdwuGVzn339aKBRre1NvC9rUwSbL8v4GLwWj5TZBRP6RvLDnF): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101721320 (571h36m35.972804355s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101848852 (571h36m35.972931887s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101854173 (571h36m35.972937208s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101868661 (571h36m35.972951706s) [seg_pos={0 0}, upload=501136]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:05:36.101932713 (571h36m35.973015748s) [upload=501136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:05:36.101937271 (571h36m35.973020316s) [upload=501136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:05:36.403553304 (571h36m36.274636349s) [upload=501136]: Done uploading segments... Dec 29 11:05:36.403573943 (571h36m36.274656978s) [upload=501136]: Upload finished. err: <nil> Duration: 35.866322397s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X...X.X.X....X...XXX......X...X....X...X..........X....X.] duration: 5.000800715s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:06:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000816867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..................................X] duration: 1m0.003690047s trace: https://jaeger.dev.storj.io/trace/2a822012276bcc5d FAIL (2025-12-29 11:06:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882470730 (571h36m32.753553775s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882771553 (571h36m32.753854588s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883107963 (571h36m32.754190998s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883533073 (571h36m32.754616118s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883944547 (571h36m32.755027582s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.884326042 (571h36m32.755409077s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886067960 (571h36m32.757150995s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886589914 (571h36m32.757672949s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886895325 (571h36m32.75797837s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888604990 (571h36m32.759688035s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888877790 (571h36m32.759960925s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.890542230 (571h36m32.761625275s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.893184912 (571h36m32.764267947s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.896004932 (571h36m32.767087967s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.901393461 (571h36m32.772476507s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:34.200413331 (571h36m34.071496366s) [upload=501136]: Next returned nil segment Dec 29 11:05:34.200437376 (571h36m34.071520411s) [upload=501136]: Waiting for error group managing segments... Dec 29 11:05:34.531472608 (571h36m34.402555643s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.589291783 (571h36m34.460374819s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.782400973 (571h36m34.653484008s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.895054052 (571h36m34.766137097s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.965988150 (571h36m34.837071195s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.068876977 (571h36m34.939960012s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.075857198 (571h36m34.946940243s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.237630089 (571h36m35.108713134s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.371788215 (571h36m35.24287126s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.635729582 (571h36m35.506812627s) [noise=true , address=145.79.227.30:4784 , piece_id=BUNTKFESZUDAYXSOIMKUG7DVUF65CQJAHDB7NVHII4A5LEWA4K4Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.650702903 (571h36m35.521785948s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.721297946 (571h36m35.592380991s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.812319244 (571h36m35.683402279s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101131227 (571h36m35.972214262s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101172385 (571h36m35.97225542s) [seg_pos={0 0}, upload=501136]: Segment reached optimal threshold of 14 pieces. Dec 29 11:05:36.101312391 (571h36m35.972395436s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101318363 (571h36m35.972401408s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101621961 (571h36m35.972704996s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection Dec 29 11:05:36.101629084 (571h36m35.972712119s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101715780 (571h36m35.972798815s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DdwuGVzn339aKBRre1NvC9rUwSbL8v4GLwWj5TZBRP6RvLDnF): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101721320 (571h36m35.972804355s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101848852 (571h36m35.972931887s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101854173 (571h36m35.972937208s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101868661 (571h36m35.972951706s) [seg_pos={0 0}, upload=501136]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:05:36.101932713 (571h36m35.973015748s) [upload=501136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:05:36.101937271 (571h36m35.973020316s) [upload=501136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:05:36.403553304 (571h36m36.274636349s) [upload=501136]: Done uploading segments... Dec 29 11:05:36.403573943 (571h36m36.274656978s) [upload=501136]: Upload finished. err: <nil> Duration: 1m0.003710716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X...] duration: 8.273911857s trace: https://jaeger.dev.storj.io/trace/7d2238a6275bd677 FAIL (2025-12-29 11:06:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A) context deadline exceeded ======LOGS====== 29 11:03:50.508127894 (571h34m50.379210929s) [upload=501117]: Starting upload Dec 29 11:03:50.508193759 (571h34m50.379276794s) [upload=501117]: Done starting upload Dec 29 11:03:50.508247781 (571h34m50.379330817s) [upload=501117]: Uploading segments... Dec 29 11:03:50.508403559 (571h34m50.379486594s) [upload=501117]: Got next segment. Inline: false Dec 29 11:03:50.508446791 (571h34m50.379529826s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:03:50.508452321 (571h34m50.379535366s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:03:50.752297219 (571h34m50.623380284s) [seg_pos={0 0}, upload=501117]: Begin upload segment... Dec 29 11:03:50.752364688 (571h34m50.623447733s) [seg_pos={0 0}, upload=501117]: Done begin upload segment. Dec 29 11:03:50.752371510 (571h34m50.623454555s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.752472783 (571h34m50.623555818s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753148890 (571h34m50.624231935s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753724707 (571h34m50.624807742s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754239077 (571h34m50.625322112s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754696648 (571h34m50.625779693s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755134231 (571h34m50.626217276s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755653469 (571h34m50.626736515s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756178259 (571h34m50.627261304s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756870086 (571h34m50.627953131s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.758714989 (571h34m50.629798034s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.760587915 (571h34m50.63167096s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.762459930 (571h34m50.633542975s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.765825709 (571h34m50.636908754s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.769535333 (571h34m50.640618378s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.773230448 (571h34m50.644313483s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:52.476568754 (571h34m52.347651789s) [upload=501117]: Next returned nil segment Dec 29 11:03:52.476590304 (571h34m52.347673349s) [upload=501117]: Waiting for error group managing segments... Dec 29 11:03:52.758910775 (571h34m52.62999382s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.761129811 (571h34m52.632212846s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763235731 (571h34m52.634318766s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763413219 (571h34m52.634496264s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.766615176 (571h34m52.637698221s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.776069348 (571h34m52.647152393s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.782923830 (571h34m52.654006865s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.785247685 (571h34m52.65633073s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.787937577 (571h34m52.659020622s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.971553178 (571h34m52.842636214s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.995774510 (571h34m52.866857545s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.110311877 (571h34m52.981394912s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144497080 (571h34m53.015580126s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144523922 (571h34m53.015606967s) [seg_pos={0 0}, upload=501117]: Segment reached optimal threshold of 13 pieces. Dec 29 11:03:53.144761574 (571h34m53.01584463s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection Dec 29 11:03:53.144770341 (571h34m53.015853386s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145027480 (571h34m53.016110525s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145045004 (571h34m53.016128039s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145080662 (571h34m53.016163707s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145089649 (571h34m53.016172684s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145117121 (571h34m53.016200166s) [seg_pos={0 0}, upload=501117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:03:53.145212052 (571h34m53.016295097s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:03:53.145219456 (571h34m53.016302501s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:03:53.435016504 (571h34m53.306099549s) [upload=501117]: Done uploading segments... Dec 29 11:03:53.435035369 (571h34m53.306118404s) [upload=501117]: Upload finished. err: <nil> Duration: 8.273942755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X...X.X.X....X...XXX......X...X....X...X..........X....X] duration: 5.001072811s trace: https://jaeger.dev.storj.io/trace/619f8cbcf164cf11 FAIL (2025-12-29 11:06:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001097247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X...X.X.X....X...XXX......X...X....X...X..........X....] duration: 4.487814948s trace: https://jaeger.dev.storj.io/trace/1e097ba5793137a FAIL (2025-12-29 11:05:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.487836288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X..] duration: 5.927322948s trace: https://jaeger.dev.storj.io/trace/7d2238a6275bd677 FAIL (2025-12-29 11:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A) context deadline exceeded ======LOGS====== 29 11:03:50.508127894 (571h34m50.379210929s) [upload=501117]: Starting upload Dec 29 11:03:50.508193759 (571h34m50.379276794s) [upload=501117]: Done starting upload Dec 29 11:03:50.508247781 (571h34m50.379330817s) [upload=501117]: Uploading segments... Dec 29 11:03:50.508403559 (571h34m50.379486594s) [upload=501117]: Got next segment. Inline: false Dec 29 11:03:50.508446791 (571h34m50.379529826s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:03:50.508452321 (571h34m50.379535366s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:03:50.752297219 (571h34m50.623380284s) [seg_pos={0 0}, upload=501117]: Begin upload segment... Dec 29 11:03:50.752364688 (571h34m50.623447733s) [seg_pos={0 0}, upload=501117]: Done begin upload segment. Dec 29 11:03:50.752371510 (571h34m50.623454555s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.752472783 (571h34m50.623555818s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753148890 (571h34m50.624231935s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753724707 (571h34m50.624807742s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754239077 (571h34m50.625322112s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754696648 (571h34m50.625779693s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755134231 (571h34m50.626217276s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755653469 (571h34m50.626736515s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756178259 (571h34m50.627261304s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756870086 (571h34m50.627953131s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.758714989 (571h34m50.629798034s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.760587915 (571h34m50.63167096s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.762459930 (571h34m50.633542975s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.765825709 (571h34m50.636908754s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.769535333 (571h34m50.640618378s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.773230448 (571h34m50.644313483s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:52.476568754 (571h34m52.347651789s) [upload=501117]: Next returned nil segment Dec 29 11:03:52.476590304 (571h34m52.347673349s) [upload=501117]: Waiting for error group managing segments... Dec 29 11:03:52.758910775 (571h34m52.62999382s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.761129811 (571h34m52.632212846s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763235731 (571h34m52.634318766s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763413219 (571h34m52.634496264s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.766615176 (571h34m52.637698221s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.776069348 (571h34m52.647152393s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.782923830 (571h34m52.654006865s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.785247685 (571h34m52.65633073s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.787937577 (571h34m52.659020622s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.971553178 (571h34m52.842636214s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.995774510 (571h34m52.866857545s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.110311877 (571h34m52.981394912s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144497080 (571h34m53.015580126s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144523922 (571h34m53.015606967s) [seg_pos={0 0}, upload=501117]: Segment reached optimal threshold of 13 pieces. Dec 29 11:03:53.144761574 (571h34m53.01584463s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection Dec 29 11:03:53.144770341 (571h34m53.015853386s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145027480 (571h34m53.016110525s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145045004 (571h34m53.016128039s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145080662 (571h34m53.016163707s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145089649 (571h34m53.016172684s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145117121 (571h34m53.016200166s) [seg_pos={0 0}, upload=501117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:03:53.145212052 (571h34m53.016295097s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:03:53.145219456 (571h34m53.016302501s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:03:53.435016504 (571h34m53.306099549s) [upload=501117]: Done uploading segments... Dec 29 11:03:53.435035369 (571h34m53.306118404s) [upload=501117]: Upload finished. err: <nil> Duration: 5.927350079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X.] duration: 5.929029025s trace: https://jaeger.dev.storj.io/trace/7d2238a6275bd677 FAIL (2025-12-29 11:05:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A) context deadline exceeded ======LOGS====== 29 11:03:50.508127894 (571h34m50.379210929s) [upload=501117]: Starting upload Dec 29 11:03:50.508193759 (571h34m50.379276794s) [upload=501117]: Done starting upload Dec 29 11:03:50.508247781 (571h34m50.379330817s) [upload=501117]: Uploading segments... Dec 29 11:03:50.508403559 (571h34m50.379486594s) [upload=501117]: Got next segment. Inline: false Dec 29 11:03:50.508446791 (571h34m50.379529826s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:03:50.508452321 (571h34m50.379535366s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:03:50.752297219 (571h34m50.623380284s) [seg_pos={0 0}, upload=501117]: Begin upload segment... Dec 29 11:03:50.752364688 (571h34m50.623447733s) [seg_pos={0 0}, upload=501117]: Done begin upload segment. Dec 29 11:03:50.752371510 (571h34m50.623454555s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.752472783 (571h34m50.623555818s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753148890 (571h34m50.624231935s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753724707 (571h34m50.624807742s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754239077 (571h34m50.625322112s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754696648 (571h34m50.625779693s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755134231 (571h34m50.626217276s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755653469 (571h34m50.626736515s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756178259 (571h34m50.627261304s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756870086 (571h34m50.627953131s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.758714989 (571h34m50.629798034s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.760587915 (571h34m50.63167096s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.762459930 (571h34m50.633542975s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.765825709 (571h34m50.636908754s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.769535333 (571h34m50.640618378s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.773230448 (571h34m50.644313483s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:52.476568754 (571h34m52.347651789s) [upload=501117]: Next returned nil segment Dec 29 11:03:52.476590304 (571h34m52.347673349s) [upload=501117]: Waiting for error group managing segments... Dec 29 11:03:52.758910775 (571h34m52.62999382s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.761129811 (571h34m52.632212846s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763235731 (571h34m52.634318766s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763413219 (571h34m52.634496264s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.766615176 (571h34m52.637698221s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.776069348 (571h34m52.647152393s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.782923830 (571h34m52.654006865s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.785247685 (571h34m52.65633073s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.787937577 (571h34m52.659020622s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.971553178 (571h34m52.842636214s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.995774510 (571h34m52.866857545s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.110311877 (571h34m52.981394912s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144497080 (571h34m53.015580126s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144523922 (571h34m53.015606967s) [seg_pos={0 0}, upload=501117]: Segment reached optimal threshold of 13 pieces. Dec 29 11:03:53.144761574 (571h34m53.01584463s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection Dec 29 11:03:53.144770341 (571h34m53.015853386s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145027480 (571h34m53.016110525s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145045004 (571h34m53.016128039s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145080662 (571h34m53.016163707s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145089649 (571h34m53.016172684s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145117121 (571h34m53.016200166s) [seg_pos={0 0}, upload=501117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:03:53.145212052 (571h34m53.016295097s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:03:53.145219456 (571h34m53.016302501s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:03:53.435016504 (571h34m53.306099549s) [upload=501117]: Done uploading segments... Dec 29 11:03:53.435035369 (571h34m53.306118404s) [upload=501117]: Upload finished. err: <nil> Duration: 5.929121109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 37.327942657s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:05:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 37.32797508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X...X.X.X....X...XXX......X...X....X...X..........X...] duration: 3.454610582s trace: https://jaeger.dev.storj.io/trace/1e097ba5793137a FAIL (2025-12-29 11:04:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454642272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X...X.X.X....X...XXX......X...X....X...X..........X..] duration: 4.114048232s trace: https://jaeger.dev.storj.io/trace/1e097ba5793137a FAIL (2025-12-29 11:04:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.114100952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.X] duration: 30.00336719s trace: https://jaeger.dev.storj.io/trace/7d2238a6275bd677 FAIL (2025-12-29 11:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A) context deadline exceeded ======LOGS====== 29 11:03:50.508127894 (571h34m50.379210929s) [upload=501117]: Starting upload Dec 29 11:03:50.508193759 (571h34m50.379276794s) [upload=501117]: Done starting upload Dec 29 11:03:50.508247781 (571h34m50.379330817s) [upload=501117]: Uploading segments... Dec 29 11:03:50.508403559 (571h34m50.379486594s) [upload=501117]: Got next segment. Inline: false Dec 29 11:03:50.508446791 (571h34m50.379529826s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:03:50.508452321 (571h34m50.379535366s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:03:50.752297219 (571h34m50.623380284s) [seg_pos={0 0}, upload=501117]: Begin upload segment... Dec 29 11:03:50.752364688 (571h34m50.623447733s) [seg_pos={0 0}, upload=501117]: Done begin upload segment. Dec 29 11:03:50.752371510 (571h34m50.623454555s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.752472783 (571h34m50.623555818s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753148890 (571h34m50.624231935s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.753724707 (571h34m50.624807742s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754239077 (571h34m50.625322112s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.754696648 (571h34m50.625779693s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755134231 (571h34m50.626217276s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.755653469 (571h34m50.626736515s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756178259 (571h34m50.627261304s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.756870086 (571h34m50.627953131s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.758714989 (571h34m50.629798034s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.760587915 (571h34m50.63167096s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.762459930 (571h34m50.633542975s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.765825709 (571h34m50.636908754s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.769535333 (571h34m50.640618378s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:50.773230448 (571h34m50.644313483s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Uploading piece... Dec 29 11:03:52.476568754 (571h34m52.347651789s) [upload=501117]: Next returned nil segment Dec 29 11:03:52.476590304 (571h34m52.347673349s) [upload=501117]: Waiting for error group managing segments... Dec 29 11:03:52.758910775 (571h34m52.62999382s) [noise=true , address=145.79.227.17:4971 , piece_id=2B2NZCWTUM4P2AKIULCWUNCIMQSATIMKWVDXYCZDMPVFBS7ASTGA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.761129811 (571h34m52.632212846s) [noise=true , address=145.79.227.13:4804 , piece_id=EMJIZBVM3LKABWY47YWITXECEJNBVXYZUPCHBEBM7PMBGEH73LCA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763235731 (571h34m52.634318766s) [noise=true , address=145.79.227.39:4014 , piece_id=ODDVP3UVIWZF6N24B64DQKA7ADLYNYTWB7GEAVM4SLWW6MRMEPEQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.763413219 (571h34m52.634496264s) [noise=true , address=145.79.227.48:4301 , piece_id=HT3PJVZE3URWV23STRRFCCMJFNOKN24Y2SNWQCOQIJMNG7CN2XRQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.766615176 (571h34m52.637698221s) [noise=true , address=145.79.227.4:4185 , piece_id=Q773V4UPG757OFR7C6HM5FOMVDIZ6BZKUHTF4RG5NKAEOJ5IV3DA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.776069348 (571h34m52.647152393s) [noise=true , address=145.79.227.5:4222 , piece_id=IWIYQNV3FG33TGHZAEG5OH6AWGEMIVDIUSRLPP5KXXBW57U7Q7HA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.782923830 (571h34m52.654006865s) [noise=true , address=145.79.227.51:4411 , piece_id=H7TMY3HE4WVVUT36H5HFENBMDBQIOUT4TVIR7YYZ2I5Y66HBIGOA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.785247685 (571h34m52.65633073s) [noise=true , address=145.79.227.20:4437 , piece_id=TEA36K7RKHANVWLBYC6MF3ALEZNZJLII4P5OGEQCWDIQDUZUDFKQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.787937577 (571h34m52.659020622s) [noise=true , address=145.79.227.52:4447 , piece_id=E7YROAWPFEOXOTX7MJ6XAXVJRO4JLFOCADTSCD44Y7YSSLIITWDQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.971553178 (571h34m52.842636214s) [noise=true , address=45.91.116.7:4012 , piece_id=KFADA47IIZA3WB65CEWRDWWZV4BOWEFPIP6NGXXZWKF4VM3PU5BA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:52.995774510 (571h34m52.866857545s) [noise=true , address=45.91.116.14:4192 , piece_id=L23IZNB4AHYRVRELORFBIPW7Y7CX47YJCHTDPUFQSCW4NY67MQ4A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.110311877 (571h34m52.981394912s) [noise=true , address=45.91.116.11:4119 , piece_id=OU4TYRJFKFPU5C6DVXHP4E6GDITBCHNBBBLGP6KVF3DJDZDSEY6A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144497080 (571h34m53.015580126s) [noise=true , address=77.90.31.14:4663 , piece_id=N4H6UCX6RPFX6B6XKFAIRXJOF6YI7ZWROOQNXJAYPJI67FNMRRCQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: <nil> Dec 29 11:03:53.144523922 (571h34m53.015606967s) [seg_pos={0 0}, upload=501117]: Segment reached optimal threshold of 13 pieces. Dec 29 11:03:53.144761574 (571h34m53.01584463s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44464->77.90.31.10:4517: use of closed network connection Dec 29 11:03:53.144770341 (571h34m53.015853386s) [noise=true , address=77.90.31.10:4517 , piece_id=D2VY3XZCGYCHOHSJ5SCFBJ4BTWGFAPUKUL6NA3E2PIHZQPQQ4H3A, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145027480 (571h34m53.016110525s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145045004 (571h34m53.016128039s) [noise=true , address=45.91.116.12:4226 , piece_id=CTOIVOWY3US7OFRDDND2UMLJ3SG7F5G4YK2BC7FF7PYZ2FWGKNFQ, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145080662 (571h34m53.016163707s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 11:03:53.145089649 (571h34m53.016172684s) [noise=true , address=77.90.31.11:4554 , piece_id=3ZN7THVAIRPFKFLU2TINHJ7BJ3MAKBT2UIBEYYMFXSIQJPADPAPA, seg_pos={0 0}, upload=501117]: Long tail cancellation due to optimal threshold Dec 29 11:03:53.145117121 (571h34m53.016200166s) [seg_pos={0 0}, upload=501117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:03:53.145212052 (571h34m53.016295097s) [upload=501117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:03:53.145219456 (571h34m53.016302501s) [upload=501117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:03:53.435016504 (571h34m53.306099549s) [upload=501117]: Done uploading segments... Dec 29 11:03:53.435035369 (571h34m53.306118404s) [upload=501117]: Upload finished. err: <nil> Duration: 30.00338917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........X...X.X.X....X...XXX......X...X....X...X..........X.] duration: 4.466894362s trace: https://jaeger.dev.storj.io/trace/1e097ba5793137a FAIL (2025-12-29 11:04:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.466921174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 22.809369323s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:03:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 22.80941439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X...X.X.X....X...XXX......X...X....X...X..........X] duration: 5.001540351s trace: https://jaeger.dev.storj.io/trace/1e097ba5793137a FAIL (2025-12-29 11:03:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001563454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X.] duration: 5.967799105s trace: https://jaeger.dev.storj.io/trace/58fcca84552b4430 FAIL (2025-12-29 11:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:02:36.653536297 (571h33m36.524619332s) [upload=501101]: Starting upload Dec 29 11:02:36.653607142 (571h33m36.524690177s) [upload=501101]: Done starting upload Dec 29 11:02:36.653716481 (571h33m36.524799526s) [upload=501101]: Uploading segments... Dec 29 11:02:36.653815198 (571h33m36.524898233s) [upload=501101]: Got next segment. Inline: false Dec 29 11:02:36.653843102 (571h33m36.524926137s) [upload=501101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:02:36.653849113 (571h33m36.524932148s) [upload=501101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:02:36.899765446 (571h33m36.770848481s) [seg_pos={0 0}, upload=501101]: Begin upload segment... Dec 29 11:02:36.899812726 (571h33m36.770895771s) [seg_pos={0 0}, upload=501101]: Done begin upload segment. Dec 29 11:02:36.899827133 (571h33m36.770910178s) [noise=true , address=77.90.31.13:4627 , piece_id=4ZBSFHLDTZQZM2LTSQCTBBDEBTSQ2UO7N2SQMPTGRZVMTLNPWYVQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.899837443 (571h33m36.770920478s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.899880886 (571h33m36.770963921s) [noise=true , address=77.90.31.10:4517 , piece_id=7LB2UT52CHVWDB57EQJ6HTIGXGG64RUXQINGBYXX4UWXMMMENUXQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.900882402 (571h33m36.771965437s) [noise=true , address=145.79.227.49:4338 , piece_id=HYM4NOM4ZGXZHPSNP2PRDMPD2B4NKRUNW7DS7Z5HKEFKPVLCYKBQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901177815 (571h33m36.77226085s) [noise=true , address=145.79.227.17:4970 , piece_id=LOOYNW6UZYLJV7CDBK5FQ2B7WIYWZCXK373EFZSBKHPMSUKZPYTA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901502103 (571h33m36.772585138s) [noise=true , address=145.79.227.13:4804 , piece_id=OH7G4FCGWSLL3WWD7MOIJQVKGN674DMWPPYBBY7IPIFI7EOABG6A, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901776114 (571h33m36.77285915s) [noise=true , address=45.91.116.4:4940 , piece_id=7QPXRXDTEHGT3MNZFR3RV5DCZ2UPWLNSCXHWWBCJUWPF65W3GEBA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.902115301 (571h33m36.773198336s) [noise=true , address=145.79.227.46:4602 , piece_id=P7PSOP53VPEU3D3TIOFRSFVVE2ZHU5GEFH3K72HOLQ2XIOUUTT5A, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.902412356 (571h33m36.773495391s) [noise=true , address=145.79.227.5:4222 , piece_id=BZUWIXIQMNC3OK7KQHJE7NEPFEQMK6EPQWXQ73IBQHU5LFQBNMJQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.903334743 (571h33m36.774417778s) [noise=true , address=145.79.227.50:4374 , piece_id=Y3VQNC2H6FW3G6G4LU2M6VGKHNWYU4UJVAOXO3MH4P2FY2IERYLA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.903434062 (571h33m36.774517097s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.904413176 (571h33m36.775496211s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.904449966 (571h33m36.775533001s) [noise=true , address=77.90.31.12:4589 , piece_id=ESROZCN2SEYZ3CJVHFPU5YU73GDPELVEGRY6PPHZHUDII3A2PDOA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.905313721 (571h33m36.776396756s) [noise=true , address=145.79.227.40:4049 , piece_id=ZZSBG6L6H5NIUCCVY3XCQ7A2J7I5O3G5FT36755X3CGBRO75MVNQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.906936000 (571h33m36.778019035s) [noise=true , address=145.79.227.51:4411 , piece_id=A2ITK4Q7FPOBUERPH554K2GDYF37AL4FTSLDZ75MBDNPMTDOLUAQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.909545809 (571h33m36.780628844s) [noise=true , address=77.90.31.16:4736 , piece_id=GUJSS2SURVIQV47FPOWEBF5YQCAUOJIJWEAUNP4T5E36MM6YXPJA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:38.611750917 (571h33m38.482833952s) [upload=501101]: Next returned nil segment Dec 29 11:02:38.611769963 (571h33m38.482852998s) [upload=501101]: Waiting for error group managing segments... Dec 29 11:02:38.892973908 (571h33m38.764056953s) [noise=true , address=145.79.227.5:4222 , piece_id=BZUWIXIQMNC3OK7KQHJE7NEPFEQMK6EPQWXQ73IBQHU5LFQBNMJQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.896963635 (571h33m38.76804667s) [noise=true , address=145.79.227.49:4338 , piece_id=HYM4NOM4ZGXZHPSNP2PRDMPD2B4NKRUNW7DS7Z5HKEFKPVLCYKBQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.902474058 (571h33m38.773557093s) [noise=true , address=145.79.227.50:4374 , piece_id=Y3VQNC2H6FW3G6G4LU2M6VGKHNWYU4UJVAOXO3MH4P2FY2IERYLA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.904919665 (571h33m38.7760027s) [noise=true , address=145.79.227.40:4049 , piece_id=ZZSBG6L6H5NIUCCVY3XCQ7A2J7I5O3G5FT36755X3CGBRO75MVNQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.905493838 (571h33m38.776576873s) [noise=true , address=145.79.227.51:4411 , piece_id=A2ITK4Q7FPOBUERPH554K2GDYF37AL4FTSLDZ75MBDNPMTDOLUAQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.906385616 (571h33m38.777468661s) [noise=true , address=145.79.227.46:4602 , piece_id=P7PSOP53VPEU3D3TIOFRSFVVE2ZHU5GEFH3K72HOLQ2XIOUUTT5A, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.908958415 (571h33m38.78004146s) [noise=true , address=145.79.227.17:4970 , piece_id=LOOYNW6UZYLJV7CDBK5FQ2B7WIYWZCXK373EFZSBKHPMSUKZPYTA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.918206665 (571h33m38.7892897s) [noise=true , address=145.79.227.13:4804 , piece_id=OH7G4FCGWSLL3WWD7MOIJQVKGN674DMWPPYBBY7IPIFI7EOABG6A, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.090720748 (571h33m38.961803793s) [noise=true , address=77.90.31.16:4736 , piece_id=GUJSS2SURVIQV47FPOWEBF5YQCAUOJIJWEAUNP4T5E36MM6YXPJA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.192630470 (571h33m39.063713515s) [noise=true , address=77.90.31.12:4589 , piece_id=ESROZCN2SEYZ3CJVHFPU5YU73GDPELVEGRY6PPHZHUDII3A2PDOA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.204909280 (571h33m39.075992315s) [noise=true , address=77.90.31.10:4517 , piece_id=7LB2UT52CHVWDB57EQJ6HTIGXGG64RUXQINGBYXX4UWXMMMENUXQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.205898995 (571h33m39.07698204s) [noise=true , address=77.90.31.13:4627 , piece_id=4ZBSFHLDTZQZM2LTSQCTBBDEBTSQ2UO7N2SQMPTGRZVMTLNPWYVQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.258582942 (571h33m39.129665987s) [noise=true , address=45.91.116.4:4940 , piece_id=7QPXRXDTEHGT3MNZFR3RV5DCZ2UPWLNSCXHWWBCJUWPF65W3GEBA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.258609683 (571h33m39.129692718s) [seg_pos={0 0}, upload=501101]: Segment reached optimal threshold of 13 pieces. Dec 29 11:02:39.258845773 (571h33m39.129928808s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 11:02:39.258852295 (571h33m39.12993533s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259123823 (571h33m39.130206858s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 11:02:39.259134322 (571h33m39.130217387s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259450144 (571h33m39.130533189s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: read tcp 188.166.225.170:52252->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 11:02:39.259474380 (571h33m39.130557445s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259516390 (571h33m39.130599435s) [seg_pos={0 0}, upload=501101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:02:39.259688798 (571h33m39.130771833s) [upload=501101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:02:39.259696793 (571h33m39.130779838s) [upload=501101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:02:39.546867160 (571h33m39.417950205s) [upload=501101]: Done uploading segments... Dec 29 11:02:39.546891605 (571h33m39.417974641s) [upload=501101]: Upload finished. err: <nil> Duration: 5.967823801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....X] duration: 30.003931103s trace: https://jaeger.dev.storj.io/trace/58fcca84552b4430 FAIL (2025-12-29 11:03:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:02:36.653536297 (571h33m36.524619332s) [upload=501101]: Starting upload Dec 29 11:02:36.653607142 (571h33m36.524690177s) [upload=501101]: Done starting upload Dec 29 11:02:36.653716481 (571h33m36.524799526s) [upload=501101]: Uploading segments... Dec 29 11:02:36.653815198 (571h33m36.524898233s) [upload=501101]: Got next segment. Inline: false Dec 29 11:02:36.653843102 (571h33m36.524926137s) [upload=501101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:02:36.653849113 (571h33m36.524932148s) [upload=501101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:02:36.899765446 (571h33m36.770848481s) [seg_pos={0 0}, upload=501101]: Begin upload segment... Dec 29 11:02:36.899812726 (571h33m36.770895771s) [seg_pos={0 0}, upload=501101]: Done begin upload segment. Dec 29 11:02:36.899827133 (571h33m36.770910178s) [noise=true , address=77.90.31.13:4627 , piece_id=4ZBSFHLDTZQZM2LTSQCTBBDEBTSQ2UO7N2SQMPTGRZVMTLNPWYVQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.899837443 (571h33m36.770920478s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.899880886 (571h33m36.770963921s) [noise=true , address=77.90.31.10:4517 , piece_id=7LB2UT52CHVWDB57EQJ6HTIGXGG64RUXQINGBYXX4UWXMMMENUXQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.900882402 (571h33m36.771965437s) [noise=true , address=145.79.227.49:4338 , piece_id=HYM4NOM4ZGXZHPSNP2PRDMPD2B4NKRUNW7DS7Z5HKEFKPVLCYKBQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901177815 (571h33m36.77226085s) [noise=true , address=145.79.227.17:4970 , piece_id=LOOYNW6UZYLJV7CDBK5FQ2B7WIYWZCXK373EFZSBKHPMSUKZPYTA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901502103 (571h33m36.772585138s) [noise=true , address=145.79.227.13:4804 , piece_id=OH7G4FCGWSLL3WWD7MOIJQVKGN674DMWPPYBBY7IPIFI7EOABG6A, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.901776114 (571h33m36.77285915s) [noise=true , address=45.91.116.4:4940 , piece_id=7QPXRXDTEHGT3MNZFR3RV5DCZ2UPWLNSCXHWWBCJUWPF65W3GEBA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.902115301 (571h33m36.773198336s) [noise=true , address=145.79.227.46:4602 , piece_id=P7PSOP53VPEU3D3TIOFRSFVVE2ZHU5GEFH3K72HOLQ2XIOUUTT5A, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.902412356 (571h33m36.773495391s) [noise=true , address=145.79.227.5:4222 , piece_id=BZUWIXIQMNC3OK7KQHJE7NEPFEQMK6EPQWXQ73IBQHU5LFQBNMJQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.903334743 (571h33m36.774417778s) [noise=true , address=145.79.227.50:4374 , piece_id=Y3VQNC2H6FW3G6G4LU2M6VGKHNWYU4UJVAOXO3MH4P2FY2IERYLA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.903434062 (571h33m36.774517097s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.904413176 (571h33m36.775496211s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.904449966 (571h33m36.775533001s) [noise=true , address=77.90.31.12:4589 , piece_id=ESROZCN2SEYZ3CJVHFPU5YU73GDPELVEGRY6PPHZHUDII3A2PDOA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.905313721 (571h33m36.776396756s) [noise=true , address=145.79.227.40:4049 , piece_id=ZZSBG6L6H5NIUCCVY3XCQ7A2J7I5O3G5FT36755X3CGBRO75MVNQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.906936000 (571h33m36.778019035s) [noise=true , address=145.79.227.51:4411 , piece_id=A2ITK4Q7FPOBUERPH554K2GDYF37AL4FTSLDZ75MBDNPMTDOLUAQ, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:36.909545809 (571h33m36.780628844s) [noise=true , address=77.90.31.16:4736 , piece_id=GUJSS2SURVIQV47FPOWEBF5YQCAUOJIJWEAUNP4T5E36MM6YXPJA, seg_pos={0 0}, upload=501101]: Uploading piece... Dec 29 11:02:38.611750917 (571h33m38.482833952s) [upload=501101]: Next returned nil segment Dec 29 11:02:38.611769963 (571h33m38.482852998s) [upload=501101]: Waiting for error group managing segments... Dec 29 11:02:38.892973908 (571h33m38.764056953s) [noise=true , address=145.79.227.5:4222 , piece_id=BZUWIXIQMNC3OK7KQHJE7NEPFEQMK6EPQWXQ73IBQHU5LFQBNMJQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.896963635 (571h33m38.76804667s) [noise=true , address=145.79.227.49:4338 , piece_id=HYM4NOM4ZGXZHPSNP2PRDMPD2B4NKRUNW7DS7Z5HKEFKPVLCYKBQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.902474058 (571h33m38.773557093s) [noise=true , address=145.79.227.50:4374 , piece_id=Y3VQNC2H6FW3G6G4LU2M6VGKHNWYU4UJVAOXO3MH4P2FY2IERYLA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.904919665 (571h33m38.7760027s) [noise=true , address=145.79.227.40:4049 , piece_id=ZZSBG6L6H5NIUCCVY3XCQ7A2J7I5O3G5FT36755X3CGBRO75MVNQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.905493838 (571h33m38.776576873s) [noise=true , address=145.79.227.51:4411 , piece_id=A2ITK4Q7FPOBUERPH554K2GDYF37AL4FTSLDZ75MBDNPMTDOLUAQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.906385616 (571h33m38.777468661s) [noise=true , address=145.79.227.46:4602 , piece_id=P7PSOP53VPEU3D3TIOFRSFVVE2ZHU5GEFH3K72HOLQ2XIOUUTT5A, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.908958415 (571h33m38.78004146s) [noise=true , address=145.79.227.17:4970 , piece_id=LOOYNW6UZYLJV7CDBK5FQ2B7WIYWZCXK373EFZSBKHPMSUKZPYTA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:38.918206665 (571h33m38.7892897s) [noise=true , address=145.79.227.13:4804 , piece_id=OH7G4FCGWSLL3WWD7MOIJQVKGN674DMWPPYBBY7IPIFI7EOABG6A, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.090720748 (571h33m38.961803793s) [noise=true , address=77.90.31.16:4736 , piece_id=GUJSS2SURVIQV47FPOWEBF5YQCAUOJIJWEAUNP4T5E36MM6YXPJA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.192630470 (571h33m39.063713515s) [noise=true , address=77.90.31.12:4589 , piece_id=ESROZCN2SEYZ3CJVHFPU5YU73GDPELVEGRY6PPHZHUDII3A2PDOA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.204909280 (571h33m39.075992315s) [noise=true , address=77.90.31.10:4517 , piece_id=7LB2UT52CHVWDB57EQJ6HTIGXGG64RUXQINGBYXX4UWXMMMENUXQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.205898995 (571h33m39.07698204s) [noise=true , address=77.90.31.13:4627 , piece_id=4ZBSFHLDTZQZM2LTSQCTBBDEBTSQ2UO7N2SQMPTGRZVMTLNPWYVQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.258582942 (571h33m39.129665987s) [noise=true , address=45.91.116.4:4940 , piece_id=7QPXRXDTEHGT3MNZFR3RV5DCZ2UPWLNSCXHWWBCJUWPF65W3GEBA, seg_pos={0 0}, upload=501101]: Done uploading piece. err: <nil> Dec 29 11:02:39.258609683 (571h33m39.129692718s) [seg_pos={0 0}, upload=501101]: Segment reached optimal threshold of 13 pieces. Dec 29 11:02:39.258845773 (571h33m39.129928808s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 11:02:39.258852295 (571h33m39.12993533s) [noise=true , address=77.90.31.14:4663 , piece_id=5U2CTJO6G4ZNFLUWUTIW5BHUPDK7QXS2TBAA5N3O3DZRK2QAFRBQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259123823 (571h33m39.130206858s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 11:02:39.259134322 (571h33m39.130217387s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ27PVOB2UVCXBRGLT3DHRGWRHTLQENT2SYCOGUXIHQMPSX6PXOQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259450144 (571h33m39.130533189s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: read tcp 188.166.225.170:52252->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 11:02:39.259474380 (571h33m39.130557445s) [noise=true , address=45.91.116.9:4262 , piece_id=4DTEXT6IYDNUILTASJN6X5OS6W4YZJ7XIPH75HXGD2A5CFD3BJAQ, seg_pos={0 0}, upload=501101]: Long tail cancellation due to optimal threshold Dec 29 11:02:39.259516390 (571h33m39.130599435s) [seg_pos={0 0}, upload=501101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:02:39.259688798 (571h33m39.130771833s) [upload=501101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:02:39.259696793 (571h33m39.130779838s) [upload=501101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:02:39.546867160 (571h33m39.417950205s) [upload=501101]: Done uploading segments... Dec 29 11:02:39.546891605 (571h33m39.417974641s) [upload=501101]: Upload finished. err: <nil> Duration: 30.003953187s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..........X...X.X.X....X...XXX......X...X....X...X..........] duration: 4.963660064s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:03:04 UTC) orbiter warning: context deadline exceeded Duration: 4.963691103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 25.681162585s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:02:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 25.68126458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........X...X.X.X....X...XXX......X...X....X...X.........] duration: 4.467607212s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:02:20 UTC) orbiter warning: context deadline exceeded Duration: 4.467633684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.....] duration: 5.837531929s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 11:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 5.837562065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 44.213353703s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:02:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 44.213369333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............X...X.X.X....X...XXX......X...X....X...X........] duration: 5.001422694s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:01:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001440549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 39.190048965s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:01:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 39.190064975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.............X...X.X.X....X...XXX......X...X....X...X.......] duration: 5.001798584s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:01:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001826977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X....] duration: 7.605904892s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 11:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 7.605934959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..............X...X.X.X....X...XXX......X...X....X...X......] duration: 5.001693954s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:00:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001722518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 52.951595103s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 11:00:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 52.95171473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X...] duration: 6.07916526s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 11:00:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 6.079201149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X..............X...X.X.X....X...XXX......X...X....X...X.....] duration: 5.000825191s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 11:00:05 UTC) orbiter warning: context deadline exceeded Duration: 5.000852374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 40.375534708s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:59:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 40.375552843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X..] duration: 6.199708474s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 10:59:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 6.199735416s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X..............X...X.X.X....X...XXX......X...X....X...X....] duration: 4.865679755s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 10:59:29 UTC) orbiter warning: context deadline exceeded Duration: 4.865696878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X.] duration: 5.981657225s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 10:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 5.981683243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 33.835785826s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:59:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 33.835812106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..X] duration: 30.003506418s trace: https://jaeger.dev.storj.io/trace/79e84d0284e12b96 FAIL (2025-12-29 10:58:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:58:25.840660366 (571h29m25.711743402s) [upload=501036]: Starting upload Dec 29 10:58:25.840718347 (571h29m25.711801392s) [upload=501036]: Done starting upload Dec 29 10:58:25.840790444 (571h29m25.71187348s) [upload=501036]: Uploading segments... Dec 29 10:58:25.840900183 (571h29m25.711983228s) [upload=501036]: Got next segment. Inline: false Dec 29 10:58:25.840941772 (571h29m25.712024807s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:58:25.840947082 (571h29m25.712030127s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:58:26.081510312 (571h29m25.952593367s) [seg_pos={0 0}, upload=501036]: Begin upload segment... Dec 29 10:58:26.081605463 (571h29m25.952688508s) [seg_pos={0 0}, upload=501036]: Done begin upload segment. Dec 29 10:58:26.081703259 (571h29m25.952786294s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.081721103 (571h29m25.952804148s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082522860 (571h29m25.953605905s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.082656374 (571h29m25.953739419s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083223373 (571h29m25.954306418s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083254072 (571h29m25.954337107s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083785634 (571h29m25.954868669s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.083919319 (571h29m25.955002354s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084249688 (571h29m25.955332723s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.084530944 (571h29m25.955613979s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085098244 (571h29m25.956181279s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.085736879 (571h29m25.956819924s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.086756942 (571h29m25.957839987s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087156853 (571h29m25.958239898s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.087599567 (571h29m25.958682612s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:26.088210760 (571h29m25.959293805s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Uploading piece... Dec 29 10:58:27.695985608 (571h29m27.567068653s) [upload=501036]: Next returned nil segment Dec 29 10:58:27.696009934 (571h29m27.567092969s) [upload=501036]: Waiting for error group managing segments... Dec 29 10:58:28.072573248 (571h29m27.943656283s) [noise=true , address=145.79.227.3:4150 , piece_id=H4PEFDU3GAALAAOYJYIHLQN6IXOAYAZ3SWRNRGXG4RQXL6URXYZA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.075927635 (571h29m27.94701067s) [noise=true , address=145.79.227.16:4911 , piece_id=W2LAGRXA5MQH3OMJ2IROJ3E5OZEFGCP4UGEA2KP7SHNHBAMLSTLA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076207077 (571h29m27.947290123s) [noise=true , address=145.79.227.7:4293 , piece_id=RTKLMCLNHM5WM73EHF4BVEEGEGPL2VUXNN62MR2AOIFSFHXYRHFA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.076383453 (571h29m27.947466488s) [noise=true , address=145.79.227.41:4087 , piece_id=NKHHHNERWIIZR7XBJTYRAK7POQDEKKL7Y5453R75L6ZGBCNDVKSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.077430557 (571h29m27.948513592s) [noise=true , address=145.79.227.19:4078 , piece_id=67TTKN5STFMVB46PY3OWDT5WP4K26J5KDQ3JKZECUCNCPZTGY3QA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.078210843 (571h29m27.949293878s) [noise=true , address=145.79.227.36:4517 , piece_id=Z3ZHDDR2Z5AQQMBWEYDVZ42U6UPK6DZ3OXSTDCE4VMY572AYXKCQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.082021108 (571h29m27.953104143s) [noise=true , address=145.79.227.51:4410 , piece_id=63MVMTX7CIXZY75LUIXFNTYG5JMGZOWDLXIHNQQBCSRXUBCA4CMQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.084807704 (571h29m27.955890739s) [noise=true , address=145.79.227.47:4265 , piece_id=ZVYELYE5534ZFDEO4TDZHDVOVTAFXHG7MXZEXIO74GWZ6N2IKUSA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.091908185 (571h29m27.96299122s) [noise=true , address=145.79.227.39:4013 , piece_id=23EKNTSQMLBEUQCOQHN4GRS7TNMQKEYOX36FEMZZR3OPPVKNMIFQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.254085986 (571h29m28.125169061s) [noise=true , address=77.90.31.21:4877 , piece_id=ICRP547GMTXQA3HGFAVUMKPA4JRRCBRFPYPMAEZSAAUSWGYAQE6Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.369098908 (571h29m28.240181953s) [noise=true , address=77.90.31.18:4808 , piece_id=EISNL7FUCGBNSSCU24OMNAKRIRRBNJYQYB562GNUZO2XGNXGIZTA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.374730120 (571h29m28.245813155s) [noise=true , address=77.90.31.13:4629 , piece_id=NC7VW6LVO7Y3R5VH4PTQCI3KJPA37LAMRF445HP6SQ6DS6FLEJXQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389396968 (571h29m28.260480003s) [noise=true , address=77.90.31.14:4664 , piece_id=AN62TW74OO5TPY67ET4QMJ6OBUQ6UQLGBC26DOSVGFFZ46GCVPKA, seg_pos={0 0}, upload=501036]: Done uploading piece. err: <nil> Dec 29 10:58:28.389417748 (571h29m28.260500783s) [seg_pos={0 0}, upload=501036]: Segment reached optimal threshold of 13 pieces. Dec 29 10:58:28.389695136 (571h29m28.260778171s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389701338 (571h29m28.260784373s) [noise=true , address=45.91.116.8:4048 , piece_id=7RRM6B3XSE7QBRYL67ZE5NZZTXT7CWGNNMTIRYTYUNBDBDUA2F7Q, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389887622 (571h29m28.260970657s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389912460 (571h29m28.260995505s) [noise=true , address=45.91.116.5:4298 , piece_id=CBHEA7LD2IXWYOY42KRTJRUNEJU5QLTU5DW33FHSXDQW5OP3UAWQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.389970982 (571h29m28.261054017s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 10:58:28.389978225 (571h29m28.26106126s) [noise=true , address=145.79.227.34:4907 , piece_id=V2ATAHIOXO7YQWZMOKJGYEWVPHIAFQTGYUVUSH4GKQL5WDAEWKBQ, seg_pos={0 0}, upload=501036]: Long tail cancellation due to optimal threshold Dec 29 10:58:28.390026757 (571h29m28.261109792s) [seg_pos={0 0}, upload=501036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:58:28.390111970 (571h29m28.261195005s) [upload=501036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:58:28.390116789 (571h29m28.261199834s) [upload=501036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:58:28.678953278 (571h29m28.550036323s) [upload=501036]: Done uploading segments... Dec 29 10:58:28.678974238 (571h29m28.550057273s) [upload=501036]: Upload finished. err: <nil> Duration: 30.003532166s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X..............X...X.X.X....X...XXX......X...X....X...X...] duration: 3.961734875s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 10:58:21 UTC) orbiter warning: context deadline exceeded Duration: 3.961761595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 42.359900269s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 42.359926549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X.X.X..X..........X...X..X....XXX.X..XXX..X.X.XX....X..] duration: 6.003130223s trace: https://jaeger.dev.storj.io/trace/20b34a620bd17a9 FAIL (2025-12-29 10:57:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BQIFCZOWPRQEB35KUOQLBVKUEYMWT4G5SBB6TXZ52EUGQWDC4PFQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 7U5KCMY2XZD7VNB2BX7MIFQUQ4O4Z5IUCAP6IBK6EKIV45ZDA5GA) context deadline exceeded ======LOGS====== Dec 29 10:56:14.252822547 (571h27m14.123905582s) [upload=501003]: Starting upload Dec 29 10:56:14.252893902 (571h27m14.123976938s) [upload=501003]: Done starting upload Dec 29 10:56:14.252952895 (571h27m14.12403593s) [upload=501003]: Uploading segments... Dec 29 10:56:14.253090207 (571h27m14.124173252s) [upload=501003]: Got next segment. Inline: false Dec 29 10:56:14.253123470 (571h27m14.124206505s) [upload=501003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:56:14.253127678 (571h27m14.124210723s) [upload=501003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:56:14.501983637 (571h27m14.373066682s) [seg_pos={0 0}, upload=501003]: Begin upload segment... Dec 29 10:56:14.502040355 (571h27m14.3731234s) [seg_pos={0 0}, upload=501003]: Done begin upload segment. Dec 29 10:56:14.502076163 (571h27m14.373159198s) [noise=true , address=145.79.227.42:4121 , piece_id=N6K7VGHW5VUCUZXRZXOIQKT4E3H4F5QLLQZ4NMRKI6LVQ2Q3KCCA, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.502123794 (571h27m14.373206839s) [noise=true , address=145.79.227.5:4221 , piece_id=ZAAVIG6Y4MQG42N4UOXIRVSWQ7NHADHBNZEDJKYOETECSI26H3JQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.502706654 (571h27m14.373789689s) [noise=true , address=145.79.227.50:4375 , piece_id=C5LDNHJJ655ZS6PH7HGGQSEOZU6PI3NCAIUWOQR2DYFGYNREWCJQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.502943465 (571h27m14.37402651s) [noise=true , address=145.79.227.29:4941 , piece_id=ULKRAXSHI5QQ7JKERR32NMFGTSAJP3JHF3AMBBBTJMBRYE7ZTTTQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.503211876 (571h27m14.374294911s) [noise=true , address=145.79.227.35:4483 , piece_id=YUUWTU64LC5W5UDXYGZM4NHUBXN6TEB7ENE557Q2O75CL646AEIQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.503567834 (571h27m14.374650879s) [noise=true , address=145.79.227.6:4258 , piece_id=3CMYLBV4JNAIAI4TXVDSQ3GPDV7S3TH5IQ7RSGMGH7LNVT7JSQWA, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.503908994 (571h27m14.374992039s) [noise=true , address=77.90.31.11:4553 , piece_id=RF4EBHZ73ICL5JVX6VCPIX7ENDM4BGZFBM3PUH25NAVXAOORVXWQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.504161184 (571h27m14.375244229s) [noise=true , address=45.91.116.12:4227 , piece_id=7U5KCMY2XZD7VNB2BX7MIFQUQ4O4Z5IUCAP6IBK6EKIV45ZDA5GA, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.504471665 (571h27m14.3755547s) [noise=true , address=145.79.227.47:4265 , piece_id=6FQCGDCN55YKIU6AOXOPTRBLD2CRWPND5Q4CHOI4HFMWVBMOJBKQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.504856538 (571h27m14.375939573s) [noise=true , address=145.79.227.37:4567 , piece_id=QQEMGMEFYCO623LYG7E57FHEORAD5BHRAKNK7FKX6HXVRNYUD4SQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.506098733 (571h27m14.377181829s) [noise=true , address=45.91.116.14:4189 , piece_id=BQIFCZOWPRQEB35KUOQLBVKUEYMWT4G5SBB6TXZ52EUGQWDC4PFQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.506657698 (571h27m14.377740733s) [noise=true , address=45.91.116.7:4010 , piece_id=GOULHSGWO3I4P632VFSKV3UZ7NODSWV3DOCYCDJ3WPMNIYPGLYRQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.507252460 (571h27m14.378335495s) [noise=true , address=45.91.116.4:4937 , piece_id=LJ3VPJGKKVPD24XHSKIM277O6OB5THOHRCOLWJ5DQJW2M2HGTP6A, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.507747292 (571h27m14.378830337s) [noise=true , address=145.79.227.15:4875 , piece_id=CIF4CWXIPC4EOWCUC73ZF4BZBDOJSFNEDX4RIPLLODDCDQVXIBAA, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.508204974 (571h27m14.379288009s) [noise=true , address=145.79.227.26:4653 , piece_id=H44KTZGV26AG4YSPTHNJ3CMRHF5MP465AEFPOJI4NWGVDZAL5JCQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:14.508724202 (571h27m14.379807237s) [noise=true , address=145.79.227.11:4730 , piece_id=SSPHN7JBHKDRM2YRCUO5GEX4JVDCBL7T7MB3W2FY2ANSEE2PL6LQ, seg_pos={0 0}, upload=501003]: Uploading piece... Dec 29 10:56:16.212439542 (571h27m16.083522577s) [upload=501003]: Next returned nil segment Dec 29 10:56:16.212456245 (571h27m16.08353928s) [upload=501003]: Waiting for error group managing segments... Dec 29 10:56:16.486596871 (571h27m16.357679906s) [noise=true , address=145.79.227.15:4875 , piece_id=CIF4CWXIPC4EOWCUC73ZF4BZBDOJSFNEDX4RIPLLODDCDQVXIBAA, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.495470196 (571h27m16.366553231s) [noise=true , address=145.79.227.5:4221 , piece_id=ZAAVIG6Y4MQG42N4UOXIRVSWQ7NHADHBNZEDJKYOETECSI26H3JQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.496742519 (571h27m16.367825554s) [noise=true , address=145.79.227.50:4375 , piece_id=C5LDNHJJ655ZS6PH7HGGQSEOZU6PI3NCAIUWOQR2DYFGYNREWCJQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.508372044 (571h27m16.379455089s) [noise=true , address=145.79.227.37:4567 , piece_id=QQEMGMEFYCO623LYG7E57FHEORAD5BHRAKNK7FKX6HXVRNYUD4SQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.508881013 (571h27m16.379964058s) [noise=true , address=145.79.227.47:4265 , piece_id=6FQCGDCN55YKIU6AOXOPTRBLD2CRWPND5Q4CHOI4HFMWVBMOJBKQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.511054942 (571h27m16.382137987s) [noise=true , address=145.79.227.42:4121 , piece_id=N6K7VGHW5VUCUZXRZXOIQKT4E3H4F5QLLQZ4NMRKI6LVQ2Q3KCCA, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.513257065 (571h27m16.38434011s) [noise=true , address=145.79.227.6:4258 , piece_id=3CMYLBV4JNAIAI4TXVDSQ3GPDV7S3TH5IQ7RSGMGH7LNVT7JSQWA, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.523780194 (571h27m16.394863239s) [noise=true , address=145.79.227.11:4730 , piece_id=SSPHN7JBHKDRM2YRCUO5GEX4JVDCBL7T7MB3W2FY2ANSEE2PL6LQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.526238514 (571h27m16.397321569s) [noise=true , address=145.79.227.26:4653 , piece_id=H44KTZGV26AG4YSPTHNJ3CMRHF5MP465AEFPOJI4NWGVDZAL5JCQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.529837467 (571h27m16.400920512s) [noise=true , address=145.79.227.29:4941 , piece_id=ULKRAXSHI5QQ7JKERR32NMFGTSAJP3JHF3AMBBBTJMBRYE7ZTTTQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:16.975457285 (571h27m16.84654032s) [noise=true , address=45.91.116.4:4937 , piece_id=LJ3VPJGKKVPD24XHSKIM277O6OB5THOHRCOLWJ5DQJW2M2HGTP6A, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:17.468535290 (571h27m17.339618325s) [noise=true , address=45.91.116.12:4227 , piece_id=7U5KCMY2XZD7VNB2BX7MIFQUQ4O4Z5IUCAP6IBK6EKIV45ZDA5GA, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:17.536371420 (571h27m17.407454455s) [noise=true , address=45.91.116.14:4189 , piece_id=BQIFCZOWPRQEB35KUOQLBVKUEYMWT4G5SBB6TXZ52EUGQWDC4PFQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: <nil> Dec 29 10:56:17.536389365 (571h27m17.4074724s) [seg_pos={0 0}, upload=501003]: Segment reached optimal threshold of 13 pieces. Dec 29 10:56:17.536541253 (571h27m17.407624289s) [noise=true , address=77.90.31.11:4553 , piece_id=RF4EBHZ73ICL5JVX6VCPIX7ENDM4BGZFBM3PUH25NAVXAOORVXWQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:49018->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:49018->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49018->77.90.31.11:4553: use of closed network connection Dec 29 10:56:17.536547285 (571h27m17.40763032s) [noise=true , address=77.90.31.11:4553 , piece_id=RF4EBHZ73ICL5JVX6VCPIX7ENDM4BGZFBM3PUH25NAVXAOORVXWQ, seg_pos={0 0}, upload=501003]: Long tail cancellation due to optimal threshold Dec 29 10:56:17.536602230 (571h27m17.407685265s) [noise=true , address=45.91.116.7:4010 , piece_id=GOULHSGWO3I4P632VFSKV3UZ7NODSWV3DOCYCDJ3WPMNIYPGLYRQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 10:56:17.536610375 (571h27m17.40769342s) [noise=true , address=45.91.116.7:4010 , piece_id=GOULHSGWO3I4P632VFSKV3UZ7NODSWV3DOCYCDJ3WPMNIYPGLYRQ, seg_pos={0 0}, upload=501003]: Long tail cancellation due to optimal threshold Dec 29 10:56:17.536777694 (571h27m17.407860729s) [noise=true , address=145.79.227.35:4483 , piece_id=YUUWTU64LC5W5UDXYGZM4NHUBXN6TEB7ENE557Q2O75CL646AEIQ, seg_pos={0 0}, upload=501003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 10:56:17.536784126 (571h27m17.407867181s) [noise=true , address=145.79.227.35:4483 , piece_id=YUUWTU64LC5W5UDXYGZM4NHUBXN6TEB7ENE557Q2O75CL646AEIQ, seg_pos={0 0}, upload=501003]: Long tail cancellation due to optimal threshold Dec 29 10:56:17.536799846 (571h27m17.407882882s) [seg_pos={0 0}, upload=501003]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:56:17.536858738 (571h27m17.407941773s) [upload=501003]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:56:17.536864068 (571h27m17.407947113s) [upload=501003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:56:17.835142335 (571h27m17.70622538s) [upload=501003]: Done uploading segments... Dec 29 10:56:17.835166601 (571h27m17.706249636s) [upload=501003]: Upload finished. err: <nil> Duration: 6.003159138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X..............X...X.X.X....X...XXX......X...X....X...X..] duration: 3.405963961s trace: https://jaeger.dev.storj.io/trace/5fcac1e07c580edd FAIL (2025-12-29 10:57:45 UTC) orbiter warning: context deadline exceeded Duration: 3.405990992s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)