Orbiter History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 469.845425ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:15:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.86964ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.767661274s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:15:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.767683095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X........X.X.....X...........XX....XX.........X........] duration: 5.941522315s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.941548554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.711525213s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:15:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.711554046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.740755811s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:15:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.740784446s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 683.498465ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:15:36 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 683.578167ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.881011792s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:15:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.881041269s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 471.888184ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:15:31 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 471.915596ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.................XX...............................X......] duration: 3.491399238s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491424397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.753993085s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:15:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.754019746s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 467.702665ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:15:05 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.764232ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.................XX...............................X.....] duration: 3.354100468s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:14:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.354124554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X........X.X.....X...........XX....XX.........X.......] duration: 6.037467619s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:14:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.037488559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.241383215s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:14:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241405255s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.................XX...............................X....] duration: 3.544721784s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544742464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.706266337s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:14:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.706297526s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 675.856473ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:14:27 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 675.882593ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 472.782445ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:14:27 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.809527ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.709069125s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:14:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.709095844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......X........X.X.....X...........XX....XX.........X......] duration: 5.826524538s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:14:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.826555456s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 474.257766ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:13:40 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.281431ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.161162635s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:13:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.161187403s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 696.374434ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:13:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 696.405122ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.................XX...............................X...] duration: 3.52408104s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:13:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524108984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.758338092s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.758367187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [........X........X.X.....X...........XX....XX.........X.....] duration: 5.844785362s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:13:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.844809056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.951998421s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:13:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.952053427s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 692.480669ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:12:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.505206ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 475.293929ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:12:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.322674ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.........X........X.X.....X...........XX....XX.........X....] duration: 5.93458825s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:12:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.934612026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.857545849s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.857573912s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 692.050761ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:12:26 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.080858ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.908243264s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:12:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.908280696s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.................XX...............................X..] duration: 3.537850911s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:12:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537903722s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 465.423234ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:11:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 465.454052ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..........X........X.X.....X...........XX....XX.........X...] duration: 5.940388021s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:11:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940438547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.145686034s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:11:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.145782199s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.779256303s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.779276771s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 685.407811ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:11:22 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 685.426297ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 471.063815ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:10:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 471.106766ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.................XX...............................X.] duration: 3.480896148s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:10:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480958308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.62114347s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:10:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.621170823s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...........X........X.X.....X...........XX....XX.........X..] duration: 5.809161037s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.809191162s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 692.347885ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 23:10:44 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.373925ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [............X........X.X.....X...........XX....XX.........X.] duration: 6.858275346s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:10:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.858380476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.765492284s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:10:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.765523433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.............X........X.X.....X...........XX....XX.........X] duration: 1m0.000697143s trace: https://jaeger.dev.storj.io/trace/28356f72adc0a38c FAIL (2025-12-08 23:10:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000724094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.................XX...............................X] duration: 5.000934536s trace: https://jaeger.dev.storj.io/trace/6fa507cece5b5879 FAIL (2025-12-08 23:10:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000957541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.144571993s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 23:09:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.144675181s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 470.318085ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 23:09:48 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.345267ms core satellite eu1

Orbiter Check History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)