Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 523374 reports (Page 5238 of 10468)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.711360486s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:03:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.711388951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.793451791s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:03:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.793480756s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.933767697s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:03:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.933798206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.549161669s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:03:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549194942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.706694618s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:02:54 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.706728572s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.845703596s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.845735338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.961622816s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:02:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.961660047s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.746643899s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.746674005s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.414754826s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:01:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414817316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.896680294s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:01:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.896706635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.046039936s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:00:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.046106011s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.498703378s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:00:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498731553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.131834307s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:00:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.131858934s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.070869163s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.070920049s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.598776313s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:00:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598810359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.735541347s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 05:59:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.735653511s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 12.744758233s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 05:59:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.74477305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.973147027s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 05:59:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 19.973164731s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.416526603s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 05:59:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.416552772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.566021233s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:59:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566054917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005315014s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 05:59:15 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005342817s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.550709776s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:58:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550755282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.383314657s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:57:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.38333764s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.582448213s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:57:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582526401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.407842212s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:56:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40787182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.526168917s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:55:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.52619715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.465487178s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:54:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46557707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.492795908s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:53:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492820254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.454540995s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:53:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454570321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.563610632s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:53:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563647964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.499236354s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:52:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499357986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.521775059s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:51:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521803886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.459210727s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:50:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459239622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.558523316s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:49:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558542262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.593638762s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:48:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.593666475s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.507680532s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:47:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507710058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.002038188s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 05:47:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002063967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.429283275s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:44:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429309966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 3.377859975s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:44:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377891505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.XX.........................................................] duration: 4.333843353s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:43:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.333873512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..XX........................................................] duration: 3.581280072s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58132124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...XX.......................................................] duration: 3.603264263s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:41:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.603296065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....XX......................................................] duration: 3.500190426s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:41:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500233818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.955809846s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:40:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.955841606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....XX.....................................................] duration: 3.366540121s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:40:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366570269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......XX....................................................] duration: 3.502516936s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:40:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502574356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.836845699s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.836875144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.760735591s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:39:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.760768935s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......XX...................................................] duration: 3.549349002s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549376664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.946892024s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:39:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946916251s core uplink us-select-1

1
...
5236
5237
5238
5239
5240
...
10468