Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.XXXXXX] duration: 5.001464483s trace: https://jaeger.dev.storj.io/trace/364250282ea6b259 FAIL (2026-01-09 02:59:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001488989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X..........X....X.....X......X.X..X...X....X.....X.] duration: 5.81006166s trace: https://jaeger.dev.storj.io/trace/730e77d196df72f0 FAIL (2026-01-09 02:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ) context deadline exceeded ======LOGS====== Jan 9 02:57:50.821384289 (827h28m50.692467374s) [upload=723792]: Starting upload Jan 9 02:57:50.821491242 (827h28m50.692574287s) [upload=723792]: Done starting upload Jan 9 02:57:50.821542200 (827h28m50.692625245s) [upload=723792]: Uploading segments... Jan 9 02:57:50.821770905 (827h28m50.69285395s) [upload=723792]: Got next segment. Inline: false Jan 9 02:57:50.821868370 (827h28m50.692951425s) [upload=723792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:57:50.821875774 (827h28m50.692958809s) [upload=723792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:57:51.064987307 (827h28m50.936070352s) [seg_pos={0 0}, upload=723792]: Begin upload segment... Jan 9 02:57:51.065043364 (827h28m50.936126409s) [seg_pos={0 0}, upload=723792]: Done begin upload segment. Jan 9 02:57:51.065076858 (827h28m50.936159893s) [noise=true , address=77.90.31.19:4841 , piece_id=YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065104059 (827h28m50.936187094s) [noise=true , address=45.91.116.7:4012 , piece_id=TL5JEEFU3J6NUIEA2XQV3TFRV6LFLY2KPCLRV6VZGBVROJFPZ2VQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065203158 (827h28m50.936286203s) [noise=true , address=45.91.116.9:4264 , piece_id=TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065966251 (827h28m50.937049286s) [noise=true , address=145.79.227.37:4565 , piece_id=FPMT4AQ4LOO5KTWEWJU7ZP66F6YMGLW2EKD4ZE6EXPZGLR36QXDQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.066389116 (827h28m50.937472161s) [noise=true , address=145.79.227.17:4970 , piece_id=VLRXSZGXUPLC35BOOHTVAEMF5CY5N2GDKCLKIWCCNSZP66RBNEYA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.066845405 (827h28m50.93792845s) [noise=true , address=145.79.227.36:4517 , piece_id=VBBZST35XSBH5QKDC4QBTIRHL363R4AICN67HDR2KMJR6JBSTZNQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.067241159 (827h28m50.938324194s) [noise=true , address=145.79.227.45:4230 , piece_id=CWEFJPKDIK4OACVKWGJLLPGV3ZTZ333MUSGSIYPOEIDLGNDPWP2Q, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.067637273 (827h28m50.938720318s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.068169887 (827h28m50.939252922s) [noise=true , address=145.79.227.16:4911 , piece_id=OXG6OMIE4URVEPIRQJNAXKVEI6SFJEM6C3S2SOS3WOTI2ZQOOM5Q, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.070102075 (827h28m50.94118512s) [noise=true , address=145.79.227.39:4013 , piece_id=PSJ4GDGKEY2VCEU5BBDCW4Q4AAPNNYDDPM52QMRHH3UER5Y5OAIQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.071498514 (827h28m50.942581549s) [noise=true , address=145.79.227.15:4876 , piece_id=OE33DVBSHJ332AH4MTRD6TUGRQYZ7WDZDDW5HGXG2O5RAKJ5JQ6A, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.074126288 (827h28m50.945209333s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.075946955 (827h28m50.94703s) [noise=true , address=145.79.227.27:4690 , piece_id=2XL6HIX6ED6RNA35Q3UXKYKFOQW75BWKS5WNRMEBI2SNULBWIQTQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.077487659 (827h28m50.948570704s) [noise=true , address=145.79.227.43:4157 , piece_id=XSGGBYLRJXKHBZMNC5U4KOXMBNZDPTMIPXEUCDWMDYUWPWM3XKVQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.078026805 (827h28m50.94910984s) [noise=true , address=145.79.227.44:4195 , piece_id=IILNNVWCAACFXTKNCHNPDOS3HJEBQVQWSVLOPY4IBSVT2X4BR3VA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.079785673 (827h28m50.950868718s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:52.787091841 (827h28m52.658174876s) [upload=723792]: Next returned nil segment Jan 9 02:57:52.787106770 (827h28m52.658189805s) [upload=723792]: Waiting for error group managing segments... Jan 9 02:57:53.060989192 (827h28m52.932072237s) [noise=true , address=145.79.227.16:4911 , piece_id=OXG6OMIE4URVEPIRQJNAXKVEI6SFJEM6C3S2SOS3WOTI2ZQOOM5Q, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.062908737 (827h28m52.933991782s) [noise=true , address=145.79.227.37:4565 , piece_id=FPMT4AQ4LOO5KTWEWJU7ZP66F6YMGLW2EKD4ZE6EXPZGLR36QXDQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.064868889 (827h28m52.935951934s) [noise=true , address=145.79.227.17:4970 , piece_id=VLRXSZGXUPLC35BOOHTVAEMF5CY5N2GDKCLKIWCCNSZP66RBNEYA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.072134824 (827h28m52.943217869s) [noise=true , address=145.79.227.36:4517 , piece_id=VBBZST35XSBH5QKDC4QBTIRHL363R4AICN67HDR2KMJR6JBSTZNQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.074892895 (827h28m52.94597593s) [noise=true , address=145.79.227.45:4230 , piece_id=CWEFJPKDIK4OACVKWGJLLPGV3ZTZ333MUSGSIYPOEIDLGNDPWP2Q, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.077771116 (827h28m52.948854161s) [noise=true , address=145.79.227.43:4157 , piece_id=XSGGBYLRJXKHBZMNC5U4KOXMBNZDPTMIPXEUCDWMDYUWPWM3XKVQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.078641323 (827h28m52.949724368s) [noise=true , address=145.79.227.44:4195 , piece_id=IILNNVWCAACFXTKNCHNPDOS3HJEBQVQWSVLOPY4IBSVT2X4BR3VA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.140712253 (827h28m53.011795298s) [noise=true , address=145.79.227.39:4013 , piece_id=PSJ4GDGKEY2VCEU5BBDCW4Q4AAPNNYDDPM52QMRHH3UER5Y5OAIQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.152820268 (827h28m53.023903313s) [noise=true , address=145.79.227.15:4876 , piece_id=OE33DVBSHJ332AH4MTRD6TUGRQYZ7WDZDDW5HGXG2O5RAKJ5JQ6A, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.258111684 (827h28m53.129194719s) [noise=true , address=145.79.227.27:4690 , piece_id=2XL6HIX6ED6RNA35Q3UXKYKFOQW75BWKS5WNRMEBI2SNULBWIQTQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.355476608 (827h28m53.226559663s) [noise=true , address=77.90.31.19:4841 , piece_id=YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.489326943 (827h28m53.360409978s) [noise=true , address=45.91.116.7:4012 , piece_id=TL5JEEFU3J6NUIEA2XQV3TFRV6LFLY2KPCLRV6VZGBVROJFPZ2VQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.813124039 (827h28m53.684207074s) [noise=true , address=45.91.116.9:4264 , piece_id=TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.813207077 (827h28m53.684290112s) [seg_pos={0 0}, upload=723792]: Segment reached optimal threshold of 13 pieces. Jan 9 02:57:53.813659198 (827h28m53.684742243s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 9 02:57:53.813668545 (827h28m53.68475158s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.813931376 (827h28m53.685014411s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Jan 9 02:57:53.813953157 (827h28m53.685036202s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.813989997 (827h28m53.685073032s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection Jan 9 02:57:53.813998193 (827h28m53.685081238s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.814033090 (827h28m53.685116125s) [seg_pos={0 0}, upload=723792]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:57:53.814232088 (827h28m53.685315133s) [upload=723792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:57:53.814239904 (827h28m53.685322949s) [upload=723792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:57:54.109494610 (827h28m53.980577645s) [upload=723792]: Done uploading segments... Jan 9 02:57:54.109513737 (827h28m53.980596772s) [upload=723792]: Upload finished. err: <nil> Duration: 5.810089073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 12.642303431s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:59:00 UTC) orbiter warning: context deadline exceeded Duration: 12.642335803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [...XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.XXXXX] duration: 5.001522359s trace: https://jaeger.dev.storj.io/trace/dd0f03b394405db FAIL (2026-01-09 02:58:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001543839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 14.693010229s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:58:47 UTC) orbiter warning: context deadline exceeded Duration: 14.693038755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 15.066739568s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:58:24 UTC) orbiter warning: context deadline exceeded Duration: 15.066769594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [....XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.XXXX] duration: 5.000525589s trace: https://jaeger.dev.storj.io/trace/618d13cd2675125e FAIL (2026-01-09 02:58:22 UTC) orbiter warning: context deadline exceeded Duration: 5.000548393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X..........X....X.....X......X.X..X...X....X.....X] duration: 30.004037112s trace: https://jaeger.dev.storj.io/trace/730e77d196df72f0 FAIL (2026-01-09 02:58:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ) context deadline exceeded ======LOGS====== Jan 9 02:57:50.821384289 (827h28m50.692467374s) [upload=723792]: Starting upload Jan 9 02:57:50.821491242 (827h28m50.692574287s) [upload=723792]: Done starting upload Jan 9 02:57:50.821542200 (827h28m50.692625245s) [upload=723792]: Uploading segments... Jan 9 02:57:50.821770905 (827h28m50.69285395s) [upload=723792]: Got next segment. Inline: false Jan 9 02:57:50.821868370 (827h28m50.692951425s) [upload=723792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:57:50.821875774 (827h28m50.692958809s) [upload=723792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:57:51.064987307 (827h28m50.936070352s) [seg_pos={0 0}, upload=723792]: Begin upload segment... Jan 9 02:57:51.065043364 (827h28m50.936126409s) [seg_pos={0 0}, upload=723792]: Done begin upload segment. Jan 9 02:57:51.065076858 (827h28m50.936159893s) [noise=true , address=77.90.31.19:4841 , piece_id=YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065104059 (827h28m50.936187094s) [noise=true , address=45.91.116.7:4012 , piece_id=TL5JEEFU3J6NUIEA2XQV3TFRV6LFLY2KPCLRV6VZGBVROJFPZ2VQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065203158 (827h28m50.936286203s) [noise=true , address=45.91.116.9:4264 , piece_id=TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.065966251 (827h28m50.937049286s) [noise=true , address=145.79.227.37:4565 , piece_id=FPMT4AQ4LOO5KTWEWJU7ZP66F6YMGLW2EKD4ZE6EXPZGLR36QXDQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.066389116 (827h28m50.937472161s) [noise=true , address=145.79.227.17:4970 , piece_id=VLRXSZGXUPLC35BOOHTVAEMF5CY5N2GDKCLKIWCCNSZP66RBNEYA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.066845405 (827h28m50.93792845s) [noise=true , address=145.79.227.36:4517 , piece_id=VBBZST35XSBH5QKDC4QBTIRHL363R4AICN67HDR2KMJR6JBSTZNQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.067241159 (827h28m50.938324194s) [noise=true , address=145.79.227.45:4230 , piece_id=CWEFJPKDIK4OACVKWGJLLPGV3ZTZ333MUSGSIYPOEIDLGNDPWP2Q, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.067637273 (827h28m50.938720318s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.068169887 (827h28m50.939252922s) [noise=true , address=145.79.227.16:4911 , piece_id=OXG6OMIE4URVEPIRQJNAXKVEI6SFJEM6C3S2SOS3WOTI2ZQOOM5Q, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.070102075 (827h28m50.94118512s) [noise=true , address=145.79.227.39:4013 , piece_id=PSJ4GDGKEY2VCEU5BBDCW4Q4AAPNNYDDPM52QMRHH3UER5Y5OAIQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.071498514 (827h28m50.942581549s) [noise=true , address=145.79.227.15:4876 , piece_id=OE33DVBSHJ332AH4MTRD6TUGRQYZ7WDZDDW5HGXG2O5RAKJ5JQ6A, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.074126288 (827h28m50.945209333s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.075946955 (827h28m50.94703s) [noise=true , address=145.79.227.27:4690 , piece_id=2XL6HIX6ED6RNA35Q3UXKYKFOQW75BWKS5WNRMEBI2SNULBWIQTQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.077487659 (827h28m50.948570704s) [noise=true , address=145.79.227.43:4157 , piece_id=XSGGBYLRJXKHBZMNC5U4KOXMBNZDPTMIPXEUCDWMDYUWPWM3XKVQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.078026805 (827h28m50.94910984s) [noise=true , address=145.79.227.44:4195 , piece_id=IILNNVWCAACFXTKNCHNPDOS3HJEBQVQWSVLOPY4IBSVT2X4BR3VA, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:51.079785673 (827h28m50.950868718s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Uploading piece... Jan 9 02:57:52.787091841 (827h28m52.658174876s) [upload=723792]: Next returned nil segment Jan 9 02:57:52.787106770 (827h28m52.658189805s) [upload=723792]: Waiting for error group managing segments... Jan 9 02:57:53.060989192 (827h28m52.932072237s) [noise=true , address=145.79.227.16:4911 , piece_id=OXG6OMIE4URVEPIRQJNAXKVEI6SFJEM6C3S2SOS3WOTI2ZQOOM5Q, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.062908737 (827h28m52.933991782s) [noise=true , address=145.79.227.37:4565 , piece_id=FPMT4AQ4LOO5KTWEWJU7ZP66F6YMGLW2EKD4ZE6EXPZGLR36QXDQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.064868889 (827h28m52.935951934s) [noise=true , address=145.79.227.17:4970 , piece_id=VLRXSZGXUPLC35BOOHTVAEMF5CY5N2GDKCLKIWCCNSZP66RBNEYA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.072134824 (827h28m52.943217869s) [noise=true , address=145.79.227.36:4517 , piece_id=VBBZST35XSBH5QKDC4QBTIRHL363R4AICN67HDR2KMJR6JBSTZNQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.074892895 (827h28m52.94597593s) [noise=true , address=145.79.227.45:4230 , piece_id=CWEFJPKDIK4OACVKWGJLLPGV3ZTZ333MUSGSIYPOEIDLGNDPWP2Q, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.077771116 (827h28m52.948854161s) [noise=true , address=145.79.227.43:4157 , piece_id=XSGGBYLRJXKHBZMNC5U4KOXMBNZDPTMIPXEUCDWMDYUWPWM3XKVQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.078641323 (827h28m52.949724368s) [noise=true , address=145.79.227.44:4195 , piece_id=IILNNVWCAACFXTKNCHNPDOS3HJEBQVQWSVLOPY4IBSVT2X4BR3VA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.140712253 (827h28m53.011795298s) [noise=true , address=145.79.227.39:4013 , piece_id=PSJ4GDGKEY2VCEU5BBDCW4Q4AAPNNYDDPM52QMRHH3UER5Y5OAIQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.152820268 (827h28m53.023903313s) [noise=true , address=145.79.227.15:4876 , piece_id=OE33DVBSHJ332AH4MTRD6TUGRQYZ7WDZDDW5HGXG2O5RAKJ5JQ6A, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.258111684 (827h28m53.129194719s) [noise=true , address=145.79.227.27:4690 , piece_id=2XL6HIX6ED6RNA35Q3UXKYKFOQW75BWKS5WNRMEBI2SNULBWIQTQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.355476608 (827h28m53.226559663s) [noise=true , address=77.90.31.19:4841 , piece_id=YHELI6F77NKQGT2OOTBRRLBONARKUPHFHLZNGZ2VVEU4Q4U4ZKCQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.489326943 (827h28m53.360409978s) [noise=true , address=45.91.116.7:4012 , piece_id=TL5JEEFU3J6NUIEA2XQV3TFRV6LFLY2KPCLRV6VZGBVROJFPZ2VQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.813124039 (827h28m53.684207074s) [noise=true , address=45.91.116.9:4264 , piece_id=TRECSFFNUY3DU3S2Y3S2YMOTK6CYQLOFFGAHDXCZZRZXHMQJMZYQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: <nil> Jan 9 02:57:53.813207077 (827h28m53.684290112s) [seg_pos={0 0}, upload=723792]: Segment reached optimal threshold of 13 pieces. Jan 9 02:57:53.813659198 (827h28m53.684742243s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 9 02:57:53.813668545 (827h28m53.68475158s) [noise=true , address=45.91.116.4:4939 , piece_id=U26XSIKWSHZTBCWQ5V3EZQIRN6XQBNROZLZIVRWZUGLRZWND5WNQ, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.813931376 (827h28m53.685014411s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Jan 9 02:57:53.813953157 (827h28m53.685036202s) [noise=true , address=77.90.31.11:4553 , piece_id=65B3N3M5OYDDT2YARNUMGZGUXHTRSCBDHZBSIKFVTOHLPLUGFSVA, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.813989997 (827h28m53.685073032s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52774->77.90.31.10:4517: use of closed network connection Jan 9 02:57:53.813998193 (827h28m53.685081238s) [noise=true , address=77.90.31.10:4517 , piece_id=3Q52TFLN6QT7YT44CTA3RXWT5UCM74UJSZQEHPYLPQFPF67ZOSJQ, seg_pos={0 0}, upload=723792]: Long tail cancellation due to optimal threshold Jan 9 02:57:53.814033090 (827h28m53.685116125s) [seg_pos={0 0}, upload=723792]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:57:53.814232088 (827h28m53.685315133s) [upload=723792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:57:53.814239904 (827h28m53.685322949s) [upload=723792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:57:54.109494610 (827h28m53.980577645s) [upload=723792]: Done uploading segments... Jan 9 02:57:54.109513737 (827h28m53.980596772s) [upload=723792]: Upload finished. err: <nil> Duration: 30.004062431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 8.404110165s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:57:57 UTC) orbiter warning: context deadline exceeded Duration: 8.404141225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 23.850431964s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:57:27 UTC) orbiter warning: context deadline exceeded Duration: 23.85046146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.....XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.XXX] duration: 5.001372451s trace: https://jaeger.dev.storj.io/trace/1b2dce00b3fa4605 FAIL (2026-01-09 02:57:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001400024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X..........X....X.....X......X.X..X...X....X.....] duration: 6.889441881s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:57:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 6.889469645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.XX] duration: 5.001483883s trace: https://jaeger.dev.storj.io/trace/43b667b43c7c479f FAIL (2026-01-09 02:56:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001504873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 14.191533911s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:56:46 UTC) orbiter warning: context deadline exceeded Duration: 14.191595819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X..........X....X.....X......X.X..X...X....X....] duration: 5.819159437s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:56:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 5.819188483s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.X] duration: 5.000594145s trace: https://jaeger.dev.storj.io/trace/218515d25e074197 FAIL (2026-01-09 02:55:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000617268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.549604406s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:55:38 UTC) orbiter warning: context deadline exceeded Duration: 7.54963814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X..........X....X.....X......X.X..X...X....X...] duration: 6.056362728s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 6.056398417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX.] duration: 3.758687502s trace: https://jaeger.dev.storj.io/trace/3eb9991d552abed4 FAIL (2026-01-09 02:54:51 UTC) orbiter warning: context deadline exceeded Duration: 3.759048408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 19.380058767s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:54:43 UTC) orbiter warning: context deadline exceeded Duration: 19.380084836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.XX] duration: 5.001017194s trace: https://jaeger.dev.storj.io/trace/3eb9991d552abed4 FAIL (2026-01-09 02:54:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001041079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X..........X....X.....X......X.X..X...X....X..] duration: 6.141708394s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:54:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 6.141725987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X..........X....X.....X......X.X..X...X....X.] duration: 6.040034454s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 6.040062818s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.273820759s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:53:25 UTC) orbiter warning: context deadline exceeded Duration: 7.273875995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.X] duration: 5.001763093s trace: https://jaeger.dev.storj.io/trace/72b9a85435f1956d FAIL (2026-01-09 02:53:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001788371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [................X..........X....X.....X......X.X..X...X....X] duration: 30.00274607s trace: https://jaeger.dev.storj.io/trace/61cb8209fbef82bc FAIL (2026-01-09 02:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ) context deadline exceeded ======LOGS====== Jan 9 02:52:24.521434209 (827h23m24.392517244s) [upload=723709]: Starting upload Jan 9 02:52:24.521574215 (827h23m24.39265725s) [upload=723709]: Done starting upload Jan 9 02:52:24.521651392 (827h23m24.392734427s) [upload=723709]: Uploading segments... Jan 9 02:52:24.521767252 (827h23m24.392850287s) [upload=723709]: Got next segment. Inline: false Jan 9 02:52:24.521866061 (827h23m24.392949096s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:52:24.521873665 (827h23m24.3929567s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:52:24.765669008 (827h23m24.636752043s) [seg_pos={0 0}, upload=723709]: Begin upload segment... Jan 9 02:52:24.765713051 (827h23m24.636796097s) [seg_pos={0 0}, upload=723709]: Done begin upload segment. Jan 9 02:52:24.765745153 (827h23m24.636828198s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.765834122 (827h23m24.636917167s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.766871106 (827h23m24.637954151s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767359356 (827h23m24.638442391s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.767747565 (827h23m24.63883061s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768181091 (827h23m24.639264126s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768209785 (827h23m24.63929283s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768707574 (827h23m24.639790609s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.768723664 (827h23m24.639806709s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769089821 (827h23m24.640172856s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769444306 (827h23m24.640527351s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.769880827 (827h23m24.640963872s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770201988 (827h23m24.641285023s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770448949 (827h23m24.641531984s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.770873337 (827h23m24.641956372s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:24.771852572 (827h23m24.642935607s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Uploading piece... Jan 9 02:52:26.466418149 (827h23m26.337501184s) [upload=723709]: Next returned nil segment Jan 9 02:52:26.466430593 (827h23m26.337513628s) [upload=723709]: Waiting for error group managing segments... Jan 9 02:52:26.745664255 (827h23m26.61674729s) [noise=true , address=145.79.227.3:4149 , piece_id=DK7ZQ3FXOVDU6LJYRVSUAYZLOGY6XUWELUYTNSO76HVXFIAYWFTQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.752024085 (827h23m26.62310712s) [noise=true , address=145.79.227.5:4223 , piece_id=UDDQ44LEBRMKAN72VTPE4NZYEHMJRTZZQNORPNMUP5GNFROBDDXQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.761521759 (827h23m26.632604794s) [noise=true , address=145.79.227.52:4446 , piece_id=HJBYUXHYAUK5V4G6MYCSSKLFRUJFVPWL63TMXZCU457XSQV7AQYA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.774202315 (827h23m26.64528538s) [noise=true , address=145.79.227.15:4876 , piece_id=N2QESW3QOBMA4SKP6K2KC4TSSGIYVBSK37LQ5TSPJPUQ3CFERHCQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.828329388 (827h23m26.699412423s) [noise=true , address=145.79.227.46:4602 , piece_id=EGMQTGCMCZY4PYKEN3HV6NQLJO5T4MVMU4LBJTTTM6SUPGO2IRFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.836950493 (827h23m26.708033538s) [noise=true , address=145.79.227.37:4567 , piece_id=NADQVBF3RGNGXKQRVYR5X3FK2DWTW4PWA6Z5UXF5YWXHXK7L2EDA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:26.837544084 (827h23m26.708627119s) [noise=true , address=145.79.227.12:4767 , piece_id=FXCKF6OWM7R5OPFPD5PS45M3LRMFNVFEAQ3GGE6SDVVLWQ6MZCWA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.057712469 (827h23m26.928795504s) [noise=true , address=77.90.31.12:4589 , piece_id=ZVDO26IGSSPGJQI6ID4EZRXS6DGOANXJRTY3U4D4PEBJSSC347VQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.101013258 (827h23m26.972096293s) [noise=true , address=77.90.31.21:4878 , piece_id=IF7X4CIOZC3DER4NKZLUDLU53GKPYYDQKUTHFM2KP7TIPFAPV4RQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.162708383 (827h23m27.033791428s) [noise=true , address=45.91.116.11:4118 , piece_id=EH5C35BSJKI52UCY7KL4QHRR3GDYZLMLGLRMUGAB3YOWFWPSM4KQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.210916155 (827h23m27.08199919s) [noise=true , address=45.91.116.9:4264 , piece_id=ECF3C3KDJ6RCHKHEXEFIVYEOHOXSZBPRE3QASHM5Q27VFFE7KLFQ, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.215535972 (827h23m27.086619007s) [noise=true , address=45.91.116.12:4227 , piece_id=KZRPXV4DJPKFOBVZ3AXJ5SXZARSK4ZTJ5ILJ7GZVS3IJGVW76Z4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243421930 (827h23m27.114504975s) [noise=true , address=45.91.116.14:4191 , piece_id=DF4BWEHZ5RZ3NSBVULE35AE66ZTU65Z2DR635PVUM2AF3SR7FNTA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: <nil> Jan 9 02:52:27.243442529 (827h23m27.114525564s) [seg_pos={0 0}, upload=723709]: Segment reached optimal threshold of 13 pieces. Jan 9 02:52:27.243602794 (827h23m27.114685829s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243608555 (827h23m27.11469159s) [noise=true , address=45.91.116.5:4298 , piece_id=HWBI6DGYDSSCLOJDCQMSGZSFJXKRZ4CPEQ3ETWANHDK36KBUJP4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243651607 (827h23m27.114734652s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Jan 9 02:52:27.243661185 (827h23m27.11474422s) [noise=true , address=45.91.116.4:4940 , piece_id=E64IOX2UGBVO3NBJNRD63RELYXUZB5PU573UXGWSJU77E2UYD5MA, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243871436 (827h23m27.114954471s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56284->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: context canceled Jan 9 02:52:27.243878539 (827h23m27.114961594s) [noise=true , address=145.79.227.4:4185 , piece_id=XPCEYDSONNQVGGQSLC6GSN7UCQGDIIB3WC5RPXC5SOP4IVEZKB4A, seg_pos={0 0}, upload=723709]: Long tail cancellation due to optimal threshold Jan 9 02:52:27.243892897 (827h23m27.114975932s) [seg_pos={0 0}, upload=723709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:52:27.243952710 (827h23m27.115035745s) [upload=723709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:52:27.243957168 (827h23m27.115040204s) [upload=723709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:52:27.533110829 (827h23m27.404193864s) [upload=723709]: Done uploading segments... Jan 9 02:52:27.533129515 (827h23m27.40421255s) [upload=723709]: Upload finished. err: <nil> Duration: 30.002770426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX.] duration: 3.791921754s trace: https://jaeger.dev.storj.io/trace/519a26c9ceb0fb97 FAIL (2026-01-09 02:52:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.791949297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.17001784s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:52:19 UTC) orbiter warning: context deadline exceeded Duration: 7.170046224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 17.392373403s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:51:53 UTC) orbiter warning: context deadline exceeded Duration: 17.392394694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X..........X....X.....X......X.X..X...X....] duration: 6.000588296s trace: https://jaeger.dev.storj.io/trace/1d1852502fd220e1 FAIL (2026-01-09 02:51:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA) context deadline exceeded ======LOGS====== 9 02:48:43.307976705 (827h19m43.17905974s) [upload=723653]: Starting upload Jan 9 02:48:43.308059893 (827h19m43.179142938s) [upload=723653]: Done starting upload Jan 9 02:48:43.308139084 (827h19m43.179222119s) [upload=723653]: Uploading segments... Jan 9 02:48:43.308310790 (827h19m43.179393835s) [upload=723653]: Got next segment. Inline: false Jan 9 02:48:43.308357319 (827h19m43.179440354s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:48:43.308363932 (827h19m43.179446977s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:48:44.742383784 (827h19m44.613466829s) [seg_pos={0 0}, upload=723653]: Begin upload segment... Jan 9 02:48:44.742433498 (827h19m44.613516543s) [seg_pos={0 0}, upload=723653]: Done begin upload segment. Jan 9 02:48:44.742462864 (827h19m44.613545899s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.742510795 (827h19m44.61359383s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743041857 (827h19m44.614124902s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743051425 (827h19m44.61413447s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743415447 (827h19m44.614498503s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743937982 (827h19m44.615021017s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744061598 (827h19m44.615144633s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744471068 (827h19m44.615554103s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744650700 (827h19m44.615733745s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745480611 (827h19m44.616563646s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745674549 (827h19m44.616757584s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.746518486 (827h19m44.617601521s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747042754 (827h19m44.618125799s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747752637 (827h19m44.618835672s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747873807 (827h19m44.618956842s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.750196149 (827h19m44.621279184s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:46.463665082 (827h19m46.334748127s) [upload=723653]: Next returned nil segment Jan 9 02:48:46.463677486 (827h19m46.334760531s) [upload=723653]: Waiting for error group managing segments... Jan 9 02:48:46.752405736 (827h19m46.623488781s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.754456250 (827h19m46.625539295s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.775466746 (827h19m46.646549781s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.818466292 (827h19m46.689549327s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.829694612 (827h19m46.700777657s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.837792301 (827h19m46.708875336s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.842686991 (827h19m46.713770036s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.123979001 (827h19m46.995062036s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.134731345 (827h19m47.0058144s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.163143024 (827h19m47.034226069s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.186245934 (827h19m47.057328989s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.290090705 (827h19m47.16117378s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739744557 (827h19m47.610827592s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739762791 (827h19m47.610845826s) [seg_pos={0 0}, upload=723653]: Segment reached optimal threshold of 13 pieces. Jan 9 02:48:47.739936652 (827h19m47.611019687s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.739942764 (827h19m47.611025799s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740089042 (827h19m47.611172077s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection Jan 9 02:48:47.740097038 (827h19m47.611180073s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740129580 (827h19m47.611212615s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.740138557 (827h19m47.611221602s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740158674 (827h19m47.61124171s) [seg_pos={0 0}, upload=723653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:48:47.740234369 (827h19m47.611317415s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:48:47.740239219 (827h19m47.611322254s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:48:49.895106881 (827h19m49.766189917s) [upload=723653]: Done uploading segments... Jan 9 02:48:49.895125757 (827h19m49.766208792s) [upload=723653]: Upload finished. err: <nil> Duration: 6.000651465s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXXX] duration: 5.001366423s trace: https://jaeger.dev.storj.io/trace/519a26c9ceb0fb97 FAIL (2026-01-09 02:51:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001390087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.593723362s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:51:07 UTC) orbiter warning: context deadline exceeded Duration: 6.593770561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXXX] duration: 5.000990643s trace: https://jaeger.dev.storj.io/trace/32995fb105f5cd9a FAIL (2026-01-09 02:50:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001017133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X..........X....X.....X......X.X..X...X...] duration: 5.845278208s trace: https://jaeger.dev.storj.io/trace/1d1852502fd220e1 FAIL (2026-01-09 02:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA) context deadline exceeded ======LOGS====== 9 02:48:43.307976705 (827h19m43.17905974s) [upload=723653]: Starting upload Jan 9 02:48:43.308059893 (827h19m43.179142938s) [upload=723653]: Done starting upload Jan 9 02:48:43.308139084 (827h19m43.179222119s) [upload=723653]: Uploading segments... Jan 9 02:48:43.308310790 (827h19m43.179393835s) [upload=723653]: Got next segment. Inline: false Jan 9 02:48:43.308357319 (827h19m43.179440354s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:48:43.308363932 (827h19m43.179446977s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:48:44.742383784 (827h19m44.613466829s) [seg_pos={0 0}, upload=723653]: Begin upload segment... Jan 9 02:48:44.742433498 (827h19m44.613516543s) [seg_pos={0 0}, upload=723653]: Done begin upload segment. Jan 9 02:48:44.742462864 (827h19m44.613545899s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.742510795 (827h19m44.61359383s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743041857 (827h19m44.614124902s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743051425 (827h19m44.61413447s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743415447 (827h19m44.614498503s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743937982 (827h19m44.615021017s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744061598 (827h19m44.615144633s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744471068 (827h19m44.615554103s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744650700 (827h19m44.615733745s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745480611 (827h19m44.616563646s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745674549 (827h19m44.616757584s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.746518486 (827h19m44.617601521s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747042754 (827h19m44.618125799s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747752637 (827h19m44.618835672s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747873807 (827h19m44.618956842s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.750196149 (827h19m44.621279184s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:46.463665082 (827h19m46.334748127s) [upload=723653]: Next returned nil segment Jan 9 02:48:46.463677486 (827h19m46.334760531s) [upload=723653]: Waiting for error group managing segments... Jan 9 02:48:46.752405736 (827h19m46.623488781s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.754456250 (827h19m46.625539295s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.775466746 (827h19m46.646549781s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.818466292 (827h19m46.689549327s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.829694612 (827h19m46.700777657s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.837792301 (827h19m46.708875336s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.842686991 (827h19m46.713770036s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.123979001 (827h19m46.995062036s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.134731345 (827h19m47.0058144s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.163143024 (827h19m47.034226069s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.186245934 (827h19m47.057328989s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.290090705 (827h19m47.16117378s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739744557 (827h19m47.610827592s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739762791 (827h19m47.610845826s) [seg_pos={0 0}, upload=723653]: Segment reached optimal threshold of 13 pieces. Jan 9 02:48:47.739936652 (827h19m47.611019687s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.739942764 (827h19m47.611025799s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740089042 (827h19m47.611172077s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection Jan 9 02:48:47.740097038 (827h19m47.611180073s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740129580 (827h19m47.611212615s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.740138557 (827h19m47.611221602s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740158674 (827h19m47.61124171s) [seg_pos={0 0}, upload=723653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:48:47.740234369 (827h19m47.611317415s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:48:47.740239219 (827h19m47.611322254s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:48:49.895106881 (827h19m49.766189917s) [upload=723653]: Done uploading segments... Jan 9 02:48:49.895125757 (827h19m49.766208792s) [upload=723653]: Upload finished. err: <nil> Duration: 5.845312734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 15.595235613s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:50:31 UTC) orbiter warning: context deadline exceeded Duration: 15.595541515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 8.817715351s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:50:07 UTC) orbiter warning: context deadline exceeded Duration: 8.81774114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXXX] duration: 5.000788639s trace: https://jaeger.dev.storj.io/trace/7af7d62fcb847fc9 FAIL (2026-01-09 02:49:53 UTC) orbiter warning: context deadline exceeded Duration: 5.000811892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X..........X....X.....X......X.X..X...X..] duration: 12.08157492s trace: https://jaeger.dev.storj.io/trace/1d1852502fd220e1 FAIL (2026-01-09 02:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA) context deadline exceeded ======LOGS====== 9 02:48:43.307976705 (827h19m43.17905974s) [upload=723653]: Starting upload Jan 9 02:48:43.308059893 (827h19m43.179142938s) [upload=723653]: Done starting upload Jan 9 02:48:43.308139084 (827h19m43.179222119s) [upload=723653]: Uploading segments... Jan 9 02:48:43.308310790 (827h19m43.179393835s) [upload=723653]: Got next segment. Inline: false Jan 9 02:48:43.308357319 (827h19m43.179440354s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:48:43.308363932 (827h19m43.179446977s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:48:44.742383784 (827h19m44.613466829s) [seg_pos={0 0}, upload=723653]: Begin upload segment... Jan 9 02:48:44.742433498 (827h19m44.613516543s) [seg_pos={0 0}, upload=723653]: Done begin upload segment. Jan 9 02:48:44.742462864 (827h19m44.613545899s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.742510795 (827h19m44.61359383s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743041857 (827h19m44.614124902s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743051425 (827h19m44.61413447s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743415447 (827h19m44.614498503s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743937982 (827h19m44.615021017s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744061598 (827h19m44.615144633s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744471068 (827h19m44.615554103s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744650700 (827h19m44.615733745s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745480611 (827h19m44.616563646s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745674549 (827h19m44.616757584s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.746518486 (827h19m44.617601521s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747042754 (827h19m44.618125799s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747752637 (827h19m44.618835672s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747873807 (827h19m44.618956842s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.750196149 (827h19m44.621279184s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:46.463665082 (827h19m46.334748127s) [upload=723653]: Next returned nil segment Jan 9 02:48:46.463677486 (827h19m46.334760531s) [upload=723653]: Waiting for error group managing segments... Jan 9 02:48:46.752405736 (827h19m46.623488781s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.754456250 (827h19m46.625539295s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.775466746 (827h19m46.646549781s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.818466292 (827h19m46.689549327s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.829694612 (827h19m46.700777657s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.837792301 (827h19m46.708875336s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.842686991 (827h19m46.713770036s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.123979001 (827h19m46.995062036s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.134731345 (827h19m47.0058144s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.163143024 (827h19m47.034226069s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.186245934 (827h19m47.057328989s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.290090705 (827h19m47.16117378s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739744557 (827h19m47.610827592s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739762791 (827h19m47.610845826s) [seg_pos={0 0}, upload=723653]: Segment reached optimal threshold of 13 pieces. Jan 9 02:48:47.739936652 (827h19m47.611019687s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.739942764 (827h19m47.611025799s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740089042 (827h19m47.611172077s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection Jan 9 02:48:47.740097038 (827h19m47.611180073s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740129580 (827h19m47.611212615s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.740138557 (827h19m47.611221602s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740158674 (827h19m47.61124171s) [seg_pos={0 0}, upload=723653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:48:47.740234369 (827h19m47.611317415s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:48:47.740239219 (827h19m47.611322254s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:48:49.895106881 (827h19m49.766189917s) [upload=723653]: Done uploading segments... Jan 9 02:48:49.895125757 (827h19m49.766208792s) [upload=723653]: Upload finished. err: <nil> Duration: 12.081600578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 12.898467224s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:49:30 UTC) orbiter warning: context deadline exceeded Duration: 12.898487844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X..........X....X.....X......X.X..X...X.] duration: 6.269654249s trace: https://jaeger.dev.storj.io/trace/1d1852502fd220e1 FAIL (2026-01-09 02:49:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA) context deadline exceeded ======LOGS====== 9 02:48:43.307976705 (827h19m43.17905974s) [upload=723653]: Starting upload Jan 9 02:48:43.308059893 (827h19m43.179142938s) [upload=723653]: Done starting upload Jan 9 02:48:43.308139084 (827h19m43.179222119s) [upload=723653]: Uploading segments... Jan 9 02:48:43.308310790 (827h19m43.179393835s) [upload=723653]: Got next segment. Inline: false Jan 9 02:48:43.308357319 (827h19m43.179440354s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:48:43.308363932 (827h19m43.179446977s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:48:44.742383784 (827h19m44.613466829s) [seg_pos={0 0}, upload=723653]: Begin upload segment... Jan 9 02:48:44.742433498 (827h19m44.613516543s) [seg_pos={0 0}, upload=723653]: Done begin upload segment. Jan 9 02:48:44.742462864 (827h19m44.613545899s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.742510795 (827h19m44.61359383s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743041857 (827h19m44.614124902s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743051425 (827h19m44.61413447s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743415447 (827h19m44.614498503s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743937982 (827h19m44.615021017s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744061598 (827h19m44.615144633s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744471068 (827h19m44.615554103s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744650700 (827h19m44.615733745s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745480611 (827h19m44.616563646s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745674549 (827h19m44.616757584s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.746518486 (827h19m44.617601521s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747042754 (827h19m44.618125799s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747752637 (827h19m44.618835672s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747873807 (827h19m44.618956842s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.750196149 (827h19m44.621279184s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:46.463665082 (827h19m46.334748127s) [upload=723653]: Next returned nil segment Jan 9 02:48:46.463677486 (827h19m46.334760531s) [upload=723653]: Waiting for error group managing segments... Jan 9 02:48:46.752405736 (827h19m46.623488781s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.754456250 (827h19m46.625539295s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.775466746 (827h19m46.646549781s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.818466292 (827h19m46.689549327s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.829694612 (827h19m46.700777657s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.837792301 (827h19m46.708875336s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.842686991 (827h19m46.713770036s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.123979001 (827h19m46.995062036s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.134731345 (827h19m47.0058144s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.163143024 (827h19m47.034226069s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.186245934 (827h19m47.057328989s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.290090705 (827h19m47.16117378s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739744557 (827h19m47.610827592s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739762791 (827h19m47.610845826s) [seg_pos={0 0}, upload=723653]: Segment reached optimal threshold of 13 pieces. Jan 9 02:48:47.739936652 (827h19m47.611019687s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.739942764 (827h19m47.611025799s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740089042 (827h19m47.611172077s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection Jan 9 02:48:47.740097038 (827h19m47.611180073s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740129580 (827h19m47.611212615s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.740138557 (827h19m47.611221602s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740158674 (827h19m47.61124171s) [seg_pos={0 0}, upload=723653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:48:47.740234369 (827h19m47.611317415s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:48:47.740239219 (827h19m47.611322254s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:48:49.895106881 (827h19m49.766189917s) [upload=723653]: Done uploading segments... Jan 9 02:48:49.895125757 (827h19m49.766208792s) [upload=723653]: Upload finished. err: <nil> Duration: 6.269670771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X..........X....X.....X......X.X..X...X] duration: 30.003807948s trace: https://jaeger.dev.storj.io/trace/1d1852502fd220e1 FAIL (2026-01-09 02:49:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA) context deadline exceeded ======LOGS====== 9 02:48:43.307976705 (827h19m43.17905974s) [upload=723653]: Starting upload Jan 9 02:48:43.308059893 (827h19m43.179142938s) [upload=723653]: Done starting upload Jan 9 02:48:43.308139084 (827h19m43.179222119s) [upload=723653]: Uploading segments... Jan 9 02:48:43.308310790 (827h19m43.179393835s) [upload=723653]: Got next segment. Inline: false Jan 9 02:48:43.308357319 (827h19m43.179440354s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:48:43.308363932 (827h19m43.179446977s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:48:44.742383784 (827h19m44.613466829s) [seg_pos={0 0}, upload=723653]: Begin upload segment... Jan 9 02:48:44.742433498 (827h19m44.613516543s) [seg_pos={0 0}, upload=723653]: Done begin upload segment. Jan 9 02:48:44.742462864 (827h19m44.613545899s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.742510795 (827h19m44.61359383s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743041857 (827h19m44.614124902s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743051425 (827h19m44.61413447s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743415447 (827h19m44.614498503s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.743937982 (827h19m44.615021017s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744061598 (827h19m44.615144633s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744471068 (827h19m44.615554103s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.744650700 (827h19m44.615733745s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745480611 (827h19m44.616563646s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.745674549 (827h19m44.616757584s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.746518486 (827h19m44.617601521s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747042754 (827h19m44.618125799s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747752637 (827h19m44.618835672s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.747873807 (827h19m44.618956842s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:44.750196149 (827h19m44.621279184s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Uploading piece... Jan 9 02:48:46.463665082 (827h19m46.334748127s) [upload=723653]: Next returned nil segment Jan 9 02:48:46.463677486 (827h19m46.334760531s) [upload=723653]: Waiting for error group managing segments... Jan 9 02:48:46.752405736 (827h19m46.623488781s) [noise=true , address=145.79.227.29:4943 , piece_id=YYQZRGLL7OG5TF7AYN46UERT5PGHB4CJKXWKSVMKSH53SRVG4CTQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.754456250 (827h19m46.625539295s) [noise=true , address=145.79.227.41:4085 , piece_id=6EPFL4PQTGBWTHECODFIJOWOVTNVABDYH76NMOXLKDVSUTFGOL2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.775466746 (827h19m46.646549781s) [noise=true , address=145.79.227.48:4302 , piece_id=5FEMF4L44XQEZXFW5ZFTTUCEQCEVOI2K22UT534JJFMLEBZHG2HA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.818466292 (827h19m46.689549327s) [noise=true , address=145.79.227.52:4445 , piece_id=7ZBPSC35XCDFMVMLSSADJPPEYXOL42AKNMQKKBA72OQMZDB5BQ3Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.829694612 (827h19m46.700777657s) [noise=true , address=145.79.227.45:4229 , piece_id=7H7KWUQF6MMRXWTMPJLVV773JVYABDNEEI2KRH7VFMNSSOPTM2EA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.837792301 (827h19m46.708875336s) [noise=true , address=145.79.227.46:4601 , piece_id=MCAFEBFUOXR42NCZS74PM5JO56GN7AMPIHZYIJXW5MWX2GU5SXOA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:46.842686991 (827h19m46.713770036s) [noise=true , address=145.79.227.25:4617 , piece_id=MCUAWBLOC272BLZKA2WLNF537KGJVQR27U6TH2KCVQ3IOWNZZPNA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.123979001 (827h19m46.995062036s) [noise=true , address=77.90.31.19:4842 , piece_id=7EJX3O6B7AFXDGXGCMXPXYDKE7RIG3XIX532P5L4GPZH7YSM7LGA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.134731345 (827h19m47.0058144s) [noise=true , address=77.90.31.21:4880 , piece_id=ICJRSZJK7JJ7CH4OBTNA7YZSOFAAZRYBSTRHFN2DT6GN5NL26KEA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.163143024 (827h19m47.034226069s) [noise=true , address=45.91.116.14:4189 , piece_id=PEHJYLMGUDXYO3TIDPPS6XOZRIHC3G4GUE3AYENMEBW2QIKZBM2Q, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.186245934 (827h19m47.057328989s) [noise=true , address=45.91.116.13:4154 , piece_id=BDR6PKX7NYUESMEP2EAE7JLXK4G7BLRRPLM2LMY7DVCSI2YQEQTA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.290090705 (827h19m47.16117378s) [noise=true , address=45.91.116.5:4298 , piece_id=XZWK43WNMY2GMY5UHMJBB37HZQ24E4ETTJKFBAEIYGS77V5LYRLA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739744557 (827h19m47.610827592s) [noise=true , address=77.90.31.11:4554 , piece_id=OUTME4KPI4KJECLDEH5NXVL5HU5JQF2AY7NEDYTDHMXQOOEOXLFA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: <nil> Jan 9 02:48:47.739762791 (827h19m47.610845826s) [seg_pos={0 0}, upload=723653]: Segment reached optimal threshold of 13 pieces. Jan 9 02:48:47.739936652 (827h19m47.611019687s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.739942764 (827h19m47.611025799s) [noise=true , address=45.91.116.9:4264 , piece_id=P2UPDWAD5YLWMOIXWF2FOX22U3J2OP4LEDVT33QCQ2Y6T34GT3UA, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740089042 (827h19m47.611172077s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4081: use of closed network connection Jan 9 02:48:47.740097038 (827h19m47.611180073s) [noise=true , address=45.91.116.10:4081 , piece_id=UBV26ATZMDH3ADBFJHESKP4SHF3Z3TKICGXJQB5UFFH6KYXCN2LQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740129580 (827h19m47.611212615s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 9 02:48:47.740138557 (827h19m47.611221602s) [noise=true , address=77.90.31.10:4520 , piece_id=JA746IT2JE7XL66PHYCBIOT7DNCJJE4UJYYHCEUYYRKBODTFVLGQ, seg_pos={0 0}, upload=723653]: Long tail cancellation due to optimal threshold Jan 9 02:48:47.740158674 (827h19m47.61124171s) [seg_pos={0 0}, upload=723653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:48:47.740234369 (827h19m47.611317415s) [upload=723653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:48:47.740239219 (827h19m47.611322254s) [upload=723653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:48:49.895106881 (827h19m49.766189917s) [upload=723653]: Done uploading segments... Jan 9 02:48:49.895125757 (827h19m49.766208792s) [upload=723653]: Upload finished. err: <nil> Duration: 30.003831253s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.X.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXXX] duration: 5.001390626s trace: https://jaeger.dev.storj.io/trace/3e5cb0bd270d8ef8 FAIL (2026-01-09 02:49:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001412738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 10.161061506s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:48:37 UTC) orbiter warning: context deadline exceeded Duration: 10.161105389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X.X.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXXX] duration: 5.001298776s trace: https://jaeger.dev.storj.io/trace/369236dc7c716c7a FAIL (2026-01-09 02:48:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001345315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X..........X....X.....X......X.X..X...] duration: 6.049298074s trace: https://jaeger.dev.storj.io/trace/44e6788d5df0ddef FAIL (2026-01-09 02:47:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 02:44:57.496598881 (827h15m57.367681926s) [upload=723599]: Starting upload Jan 9 02:44:57.496747344 (827h15m57.367830389s) [upload=723599]: Done starting upload Jan 9 02:44:57.496813850 (827h15m57.367896895s) [upload=723599]: Uploading segments... Jan 9 02:44:57.497093854 (827h15m57.368176899s) [upload=723599]: Got next segment. Inline: false Jan 9 02:44:57.497138358 (827h15m57.368221403s) [upload=723599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:44:57.497145521 (827h15m57.368228556s) [upload=723599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:44:57.747645310 (827h15m57.618728345s) [seg_pos={0 0}, upload=723599]: Begin upload segment... Jan 9 02:44:57.747702039 (827h15m57.618785074s) [seg_pos={0 0}, upload=723599]: Done begin upload segment. Jan 9 02:44:57.747777292 (827h15m57.618860327s) [noise=true , address=145.79.227.32:4834 , piece_id=GZTCNFXBKLY4GNHF7WSWAFORLIOIIQDQGQSD7LC7JWE6L2ASQYKQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.747897912 (827h15m57.618980947s) [noise=true , address=45.91.116.12:4227 , piece_id=HHG2NFUEMBL4PVBI6QB7SWRX6WVO5TJNEFUWGWAVELIVVL75TI2Q, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.748585210 (827h15m57.619668255s) [noise=true , address=45.91.116.11:4120 , piece_id=EGH35CXG5K2A4PCEC2AIG4DJAZAQRBJTK77UFD6X5DKCUFR4I6LQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.749031520 (827h15m57.620114555s) [noise=true , address=145.79.227.3:4149 , piece_id=APQH5M35K3QTLK6RUOK6LSBY5THRQ6ZLVZBU7BJ5OHHBWPBPPSTQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.750017287 (827h15m57.621100322s) [noise=true , address=145.79.227.5:4222 , piece_id=AUVHWJFCZF6KAOM2LXV4CILFSPZBTBLYM7XRQEZLHKTHRDZVBHQQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.750466993 (827h15m57.621550028s) [noise=true , address=45.91.116.9:4261 , piece_id=QQL5OQJPHLB3F2NQG2WE5OW2UUZC62ATCJQTP72EMKRFSW3ZDJFQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.751120919 (827h15m57.622203954s) [noise=true , address=77.90.31.17:4773 , piece_id=LMJ5KPUFFWF4VQI2NN7A2L5G7P6YUJG7JHSAWC3LJLKSDPDUDMYA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.751547541 (827h15m57.622630586s) [noise=true , address=77.90.31.18:4808 , piece_id=6H32WGEIDCN6MPSEPFVZ64MXP3L7WCDFJ5EQPMEUZZSP3TLJYBMA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.752457684 (827h15m57.623540719s) [noise=true , address=77.90.31.12:4590 , piece_id=HWJPAM5SVDC5WIHTX6RW3YMDGANAWT7FQE4JERN2UPIFYYZBXT3A, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.753781474 (827h15m57.624864519s) [noise=true , address=77.90.31.19:4844 , piece_id=PVWMCAAZZY25A2SV57K36VBLMW4Q64UHSSPYY2E7GOSIURUIZR7Q, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.753961297 (827h15m57.625044332s) [noise=true , address=145.79.227.37:4567 , piece_id=JUD5OGEDGCN52MXEHILSWNTVXVP4FX7NQ25TNZ32MO4KAALVF5SA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.754386135 (827h15m57.62546918s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.754971019 (827h15m57.626054064s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755225403 (827h15m57.626308448s) [noise=true , address=145.79.227.18:4006 , piece_id=PWXQDSSIL4QADJD7STZZIFUU5NFPZWASHKJLLZCYJFVNM6RN2NPQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755786432 (827h15m57.626869478s) [noise=true , address=145.79.227.15:4873 , piece_id=2AAOBCXTMSH6WHRYAMLJSQG7KZ6SERFEO4MSXVPXSRL7PBI37GWQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755939463 (827h15m57.627022509s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:59.466944564 (827h15m59.338027599s) [upload=723599]: Next returned nil segment Jan 9 02:44:59.466993378 (827h15m59.338076413s) [upload=723599]: Waiting for error group managing segments... Jan 9 02:44:59.758078134 (827h15m59.629161169s) [noise=true , address=145.79.227.18:4006 , piece_id=PWXQDSSIL4QADJD7STZZIFUU5NFPZWASHKJLLZCYJFVNM6RN2NPQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.762241231 (827h15m59.633324266s) [noise=true , address=145.79.227.15:4873 , piece_id=2AAOBCXTMSH6WHRYAMLJSQG7KZ6SERFEO4MSXVPXSRL7PBI37GWQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.763877146 (827h15m59.634960181s) [noise=true , address=145.79.227.5:4222 , piece_id=AUVHWJFCZF6KAOM2LXV4CILFSPZBTBLYM7XRQEZLHKTHRDZVBHQQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.803634238 (827h15m59.674717273s) [noise=true , address=145.79.227.37:4567 , piece_id=JUD5OGEDGCN52MXEHILSWNTVXVP4FX7NQ25TNZ32MO4KAALVF5SA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.819804568 (827h15m59.690887633s) [noise=true , address=145.79.227.32:4834 , piece_id=GZTCNFXBKLY4GNHF7WSWAFORLIOIIQDQGQSD7LC7JWE6L2ASQYKQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.835152532 (827h15m59.706235567s) [noise=true , address=145.79.227.3:4149 , piece_id=APQH5M35K3QTLK6RUOK6LSBY5THRQ6ZLVZBU7BJ5OHHBWPBPPSTQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.019930129 (827h15m59.891013164s) [noise=true , address=77.90.31.12:4590 , piece_id=HWJPAM5SVDC5WIHTX6RW3YMDGANAWT7FQE4JERN2UPIFYYZBXT3A, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.037366690 (827h15m59.908449735s) [noise=true , address=77.90.31.19:4844 , piece_id=PVWMCAAZZY25A2SV57K36VBLMW4Q64UHSSPYY2E7GOSIURUIZR7Q, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.047316836 (827h15m59.918399881s) [noise=true , address=77.90.31.17:4773 , piece_id=LMJ5KPUFFWF4VQI2NN7A2L5G7P6YUJG7JHSAWC3LJLKSDPDUDMYA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.047812931 (827h15m59.918895966s) [noise=true , address=77.90.31.18:4808 , piece_id=6H32WGEIDCN6MPSEPFVZ64MXP3L7WCDFJ5EQPMEUZZSP3TLJYBMA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.175437312 (827h16m0.046520347s) [noise=true , address=45.91.116.12:4227 , piece_id=HHG2NFUEMBL4PVBI6QB7SWRX6WVO5TJNEFUWGWAVELIVVL75TI2Q, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.184759462 (827h16m0.055842507s) [noise=true , address=45.91.116.11:4120 , piece_id=EGH35CXG5K2A4PCEC2AIG4DJAZAQRBJTK77UFD6X5DKCUFR4I6LQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.186177773 (827h16m0.057260818s) [noise=true , address=45.91.116.9:4261 , piece_id=QQL5OQJPHLB3F2NQG2WE5OW2UUZC62ATCJQTP72EMKRFSW3ZDJFQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.186187311 (827h16m0.057270346s) [seg_pos={0 0}, upload=723599]: Segment reached optimal threshold of 13 pieces. Jan 9 02:45:00.186364599 (827h16m0.057447634s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 02:45:00.186373756 (827h16m0.057456791s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186655783 (827h16m0.057738838s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jan 9 02:45:00.186681091 (827h16m0.057764126s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186743901 (827h16m0.057826936s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection Jan 9 02:45:00.186753118 (827h16m0.057836153s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186771434 (827h16m0.057854479s) [seg_pos={0 0}, upload=723599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:45:00.186836186 (827h16m0.057919231s) [upload=723599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:45:00.186840705 (827h16m0.057923751s) [upload=723599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:45:00.489612466 (827h16m0.360695521s) [upload=723599]: Done uploading segments... Jan 9 02:45:00.489635499 (827h16m0.360718544s) [upload=723599]: Upload finished. err: <nil> Duration: 6.049321167s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 10.447925429s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:47:35 UTC) orbiter warning: context deadline exceeded Duration: 10.447957751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X..........X....X.....X......X.X..X..] duration: 6.053626537s trace: https://jaeger.dev.storj.io/trace/44e6788d5df0ddef FAIL (2026-01-09 02:47:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 02:44:57.496598881 (827h15m57.367681926s) [upload=723599]: Starting upload Jan 9 02:44:57.496747344 (827h15m57.367830389s) [upload=723599]: Done starting upload Jan 9 02:44:57.496813850 (827h15m57.367896895s) [upload=723599]: Uploading segments... Jan 9 02:44:57.497093854 (827h15m57.368176899s) [upload=723599]: Got next segment. Inline: false Jan 9 02:44:57.497138358 (827h15m57.368221403s) [upload=723599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 02:44:57.497145521 (827h15m57.368228556s) [upload=723599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 02:44:57.747645310 (827h15m57.618728345s) [seg_pos={0 0}, upload=723599]: Begin upload segment... Jan 9 02:44:57.747702039 (827h15m57.618785074s) [seg_pos={0 0}, upload=723599]: Done begin upload segment. Jan 9 02:44:57.747777292 (827h15m57.618860327s) [noise=true , address=145.79.227.32:4834 , piece_id=GZTCNFXBKLY4GNHF7WSWAFORLIOIIQDQGQSD7LC7JWE6L2ASQYKQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.747897912 (827h15m57.618980947s) [noise=true , address=45.91.116.12:4227 , piece_id=HHG2NFUEMBL4PVBI6QB7SWRX6WVO5TJNEFUWGWAVELIVVL75TI2Q, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.748585210 (827h15m57.619668255s) [noise=true , address=45.91.116.11:4120 , piece_id=EGH35CXG5K2A4PCEC2AIG4DJAZAQRBJTK77UFD6X5DKCUFR4I6LQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.749031520 (827h15m57.620114555s) [noise=true , address=145.79.227.3:4149 , piece_id=APQH5M35K3QTLK6RUOK6LSBY5THRQ6ZLVZBU7BJ5OHHBWPBPPSTQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.750017287 (827h15m57.621100322s) [noise=true , address=145.79.227.5:4222 , piece_id=AUVHWJFCZF6KAOM2LXV4CILFSPZBTBLYM7XRQEZLHKTHRDZVBHQQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.750466993 (827h15m57.621550028s) [noise=true , address=45.91.116.9:4261 , piece_id=QQL5OQJPHLB3F2NQG2WE5OW2UUZC62ATCJQTP72EMKRFSW3ZDJFQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.751120919 (827h15m57.622203954s) [noise=true , address=77.90.31.17:4773 , piece_id=LMJ5KPUFFWF4VQI2NN7A2L5G7P6YUJG7JHSAWC3LJLKSDPDUDMYA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.751547541 (827h15m57.622630586s) [noise=true , address=77.90.31.18:4808 , piece_id=6H32WGEIDCN6MPSEPFVZ64MXP3L7WCDFJ5EQPMEUZZSP3TLJYBMA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.752457684 (827h15m57.623540719s) [noise=true , address=77.90.31.12:4590 , piece_id=HWJPAM5SVDC5WIHTX6RW3YMDGANAWT7FQE4JERN2UPIFYYZBXT3A, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.753781474 (827h15m57.624864519s) [noise=true , address=77.90.31.19:4844 , piece_id=PVWMCAAZZY25A2SV57K36VBLMW4Q64UHSSPYY2E7GOSIURUIZR7Q, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.753961297 (827h15m57.625044332s) [noise=true , address=145.79.227.37:4567 , piece_id=JUD5OGEDGCN52MXEHILSWNTVXVP4FX7NQ25TNZ32MO4KAALVF5SA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.754386135 (827h15m57.62546918s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.754971019 (827h15m57.626054064s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755225403 (827h15m57.626308448s) [noise=true , address=145.79.227.18:4006 , piece_id=PWXQDSSIL4QADJD7STZZIFUU5NFPZWASHKJLLZCYJFVNM6RN2NPQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755786432 (827h15m57.626869478s) [noise=true , address=145.79.227.15:4873 , piece_id=2AAOBCXTMSH6WHRYAMLJSQG7KZ6SERFEO4MSXVPXSRL7PBI37GWQ, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:57.755939463 (827h15m57.627022509s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Uploading piece... Jan 9 02:44:59.466944564 (827h15m59.338027599s) [upload=723599]: Next returned nil segment Jan 9 02:44:59.466993378 (827h15m59.338076413s) [upload=723599]: Waiting for error group managing segments... Jan 9 02:44:59.758078134 (827h15m59.629161169s) [noise=true , address=145.79.227.18:4006 , piece_id=PWXQDSSIL4QADJD7STZZIFUU5NFPZWASHKJLLZCYJFVNM6RN2NPQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.762241231 (827h15m59.633324266s) [noise=true , address=145.79.227.15:4873 , piece_id=2AAOBCXTMSH6WHRYAMLJSQG7KZ6SERFEO4MSXVPXSRL7PBI37GWQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.763877146 (827h15m59.634960181s) [noise=true , address=145.79.227.5:4222 , piece_id=AUVHWJFCZF6KAOM2LXV4CILFSPZBTBLYM7XRQEZLHKTHRDZVBHQQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.803634238 (827h15m59.674717273s) [noise=true , address=145.79.227.37:4567 , piece_id=JUD5OGEDGCN52MXEHILSWNTVXVP4FX7NQ25TNZ32MO4KAALVF5SA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.819804568 (827h15m59.690887633s) [noise=true , address=145.79.227.32:4834 , piece_id=GZTCNFXBKLY4GNHF7WSWAFORLIOIIQDQGQSD7LC7JWE6L2ASQYKQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:44:59.835152532 (827h15m59.706235567s) [noise=true , address=145.79.227.3:4149 , piece_id=APQH5M35K3QTLK6RUOK6LSBY5THRQ6ZLVZBU7BJ5OHHBWPBPPSTQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.019930129 (827h15m59.891013164s) [noise=true , address=77.90.31.12:4590 , piece_id=HWJPAM5SVDC5WIHTX6RW3YMDGANAWT7FQE4JERN2UPIFYYZBXT3A, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.037366690 (827h15m59.908449735s) [noise=true , address=77.90.31.19:4844 , piece_id=PVWMCAAZZY25A2SV57K36VBLMW4Q64UHSSPYY2E7GOSIURUIZR7Q, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.047316836 (827h15m59.918399881s) [noise=true , address=77.90.31.17:4773 , piece_id=LMJ5KPUFFWF4VQI2NN7A2L5G7P6YUJG7JHSAWC3LJLKSDPDUDMYA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.047812931 (827h15m59.918895966s) [noise=true , address=77.90.31.18:4808 , piece_id=6H32WGEIDCN6MPSEPFVZ64MXP3L7WCDFJ5EQPMEUZZSP3TLJYBMA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.175437312 (827h16m0.046520347s) [noise=true , address=45.91.116.12:4227 , piece_id=HHG2NFUEMBL4PVBI6QB7SWRX6WVO5TJNEFUWGWAVELIVVL75TI2Q, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.184759462 (827h16m0.055842507s) [noise=true , address=45.91.116.11:4120 , piece_id=EGH35CXG5K2A4PCEC2AIG4DJAZAQRBJTK77UFD6X5DKCUFR4I6LQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.186177773 (827h16m0.057260818s) [noise=true , address=45.91.116.9:4261 , piece_id=QQL5OQJPHLB3F2NQG2WE5OW2UUZC62ATCJQTP72EMKRFSW3ZDJFQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: <nil> Jan 9 02:45:00.186187311 (827h16m0.057270346s) [seg_pos={0 0}, upload=723599]: Segment reached optimal threshold of 13 pieces. Jan 9 02:45:00.186364599 (827h16m0.057447634s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 02:45:00.186373756 (827h16m0.057456791s) [noise=true , address=45.91.116.10:4083 , piece_id=NWFAOPBN4ATHTNMDKT4MPQCZBL727J5Q7ZOI2ZSIC5BBZT6737OA, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186655783 (827h16m0.057738838s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jan 9 02:45:00.186681091 (827h16m0.057764126s) [noise=true , address=45.91.116.14:4189 , piece_id=BSN6ICOIRTQLVSDBSVRYOJMJRDIVSJVU2ZRP73ECLD3CMPGECAJQ, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186743901 (827h16m0.057826936s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54438->145.79.227.48:4302: use of closed network connection Jan 9 02:45:00.186753118 (827h16m0.057836153s) [noise=true , address=145.79.227.48:4302 , piece_id=UASIR3UZNW4KGIEKCX7HN6SFEUWXJCPOP7PYYFXMVJCXSFOGWOTQ, seg_pos={0 0}, upload=723599]: Long tail cancellation due to optimal threshold Jan 9 02:45:00.186771434 (827h16m0.057854479s) [seg_pos={0 0}, upload=723599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 02:45:00.186836186 (827h16m0.057919231s) [upload=723599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 02:45:00.186840705 (827h16m0.057923751s) [upload=723599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 02:45:00.489612466 (827h16m0.360695521s) [upload=723599]: Done uploading segments... Jan 9 02:45:00.489635499 (827h16m0.360718544s) [upload=723599]: Upload finished. err: <nil> Duration: 6.053654089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.X.X.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXXX] duration: 5.000698298s trace: https://jaeger.dev.storj.io/trace/628f1dff1650e025 FAIL (2026-01-09 02:47:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000711853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 17.384780658s trace: https://jaeger.dev.storj.io/trace/7adf8a418256ad53 FAIL (2026-01-09 02:46:46 UTC) orbiter warning: context deadline exceeded Duration: 17.384808229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..X.X.XX.XX.......XX.XXX.XX.X..XXXXXX.XXXXX.X..XXXX..X..XXXX] duration: 5.00188337s trace: https://jaeger.dev.storj.io/trace/39a74457ad5167af FAIL (2026-01-09 02:46:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001911283s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)