Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.828668089s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:41:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.828688878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.989361336s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:40:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.989387356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.693744151s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:40:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.693785329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.034693519s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.034743324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................X...................X..........X.X...X.] duration: 3.371902234s trace: https://jaeger.dev.storj.io/trace/35c63dd154a0c40 FAIL (2025-12-08 23:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.371932101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..XX....XX.........X..................X................X....] duration: 5.803581097s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:40:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.803610813s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 482.291324ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:40:21 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 482.319497ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.171176716s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:40:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.171208436s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.953060081s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:40:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.953093274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.081720105s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.081751082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...XX....XX.........X..................X................X...] duration: 5.97411299s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:39:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.97414451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................X...................X..........X.X...X] duration: 5.00103248s trace: https://jaeger.dev.storj.io/trace/35c63dd154a0c40 FAIL (2025-12-08 23:39:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001053701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.831767042s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:39:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.831809893s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 493.319011ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:39:34 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 493.375058ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.768307471s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.76833384s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.11084932s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:39:19 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 2.110874738s core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.674359006s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:39:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.674382201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....XX....XX.........X..................X................X..] duration: 5.754622382s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:39:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.754654674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.885122939s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:39:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.885153959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.010765604s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:38:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.010794589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.920272461s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:38:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.920294824s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X...................X..........X.X...] duration: 3.559368322s trace: https://jaeger.dev.storj.io/trace/175d75ded74ed649 FAIL (2025-12-08 23:38:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559397529s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....XX....XX.........X..................X................X.] duration: 5.657878327s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:38:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.657904877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......XX....XX.........X..................X................X] duration: 1m0.000489998s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000515998s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 481.757326ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:38:30 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 481.786973ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.662272749s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:38:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.662302304s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X...................X..........X.X..] duration: 3.545359274s trace: https://jaeger.dev.storj.io/trace/175d75ded74ed649 FAIL (2025-12-08 23:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545389792s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 481.347135ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:38:14 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 481.375208ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.677624799s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:37:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.677654446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.996769391s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:37:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.996791582s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 475.487978ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:37:44 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.514649ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 4.952300575s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:37:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.952338477s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.737590806s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:37:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.737616214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.028232564s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:37:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.028279272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X...................X..........X.X.] duration: 3.514160343s trace: https://jaeger.dev.storj.io/trace/175d75ded74ed649 FAIL (2025-12-08 23:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514288355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......XX....XX.........X..................X................] duration: 5.881414287s trace: https://jaeger.dev.storj.io/trace/5b04940483c72074 FAIL (2025-12-08 23:37:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.881441048s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 473.883212ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:36:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 473.900666ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.31944273s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:36:52 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.319478778s core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........XX....XX.........X..................X...............] duration: 5.670217015s trace: https://jaeger.dev.storj.io/trace/5b04940483c72074 FAIL (2025-12-08 23:36:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.670616946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.824127039s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:36:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.82415356s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.619406783s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:36:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.619431491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X...................X..........X.X] duration: 5.000833817s trace: https://jaeger.dev.storj.io/trace/175d75ded74ed649 FAIL (2025-12-08 23:36:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00086077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.191618485s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:36:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.191649724s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.056000346s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:36:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.056015705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.857653836s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:36:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.85770257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.016327512s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:36:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.01635738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.730612071s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:35:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.730639825s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.729180712s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:35:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729207983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.806010719s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:35:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.806037982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........XX....XX.........X..................X..............] duration: 5.775135245s trace: https://jaeger.dev.storj.io/trace/5b04940483c72074 FAIL (2025-12-08 23:35:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.775162688s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)