Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 5.93164135s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.931672911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.479689051s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:54:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48055562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.501937237s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:54:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50196508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.757623296s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:54:16 UTC) orbiter warning: unexpected EOF Duration: 5.758045599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 5.921056968s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:54:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.921086063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.612123031s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:54:01 UTC) orbiter warning: unexpected EOF Duration: 5.612150563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.641434818s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:53:36 UTC) orbiter warning: unexpected EOF Duration: 5.64146645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.397438986s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:53:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397462379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 5.963487968s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.963513578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.627095918s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:53:07 UTC) orbiter warning: unexpected EOF Duration: 5.627128681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.662464231s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:52:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.662538833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 5.979567606s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.979630245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.650587496s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:52:26 UTC) orbiter warning: unexpected EOF Duration: 5.650605449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.447225996s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:52:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447245572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 5.869065646s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:52:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.869099511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.374404348s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:51:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.374435277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.457168499s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:51:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457211371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.711444475s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:51:12 UTC) orbiter warning: unexpected EOF Duration: 5.711480493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 6.09839911s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:50:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.09843064s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.697206199s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:50:40 UTC) orbiter warning: unexpected EOF Duration: 5.697231926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.471705256s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:50:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472802174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 5.902830933s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.902869085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.428006387s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:49:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428033467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.667102074s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:49:42 UTC) orbiter warning: unexpected EOF Duration: 5.667135528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 5.777496166s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:49:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.7775251s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.768946928s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:49:00 UTC) orbiter warning: unexpected EOF Duration: 5.769016121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.511859311s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:48:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51188514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.519013965s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:48:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519042982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 5.920605669s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:48:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.920647058s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.760450369s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:47:52 UTC) orbiter warning: unexpected EOF Duration: 5.76048255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.514389154s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:47:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514442707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 5.79768055s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.797714724s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.394083434s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:46:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.394105848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.688290188s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:46:45 UTC) orbiter warning: unexpected EOF Duration: 5.688313041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.406223945s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:46:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406255265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 5.633984324s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:46:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.634015783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.66974369s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:45:44 UTC) orbiter warning: unexpected EOF Duration: 5.669763538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 5.940923115s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:45:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.94095712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.691245683s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:45:19 UTC) orbiter warning: unexpected EOF Duration: 5.691278687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 4.325568182s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:45:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.32560404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 5.780191681s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.780247597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.751712767s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:44:28 UTC) orbiter warning: unexpected EOF Duration: 5.751805133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.44953561s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:44:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449565277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.636010789s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:43:59 UTC) orbiter warning: unexpected EOF Duration: 5.636043321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.425671873s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:43:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425697551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.838856413s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:43:24 UTC) orbiter warning: unexpected EOF Duration: 5.838888635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 5.897304572s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:43:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.897337416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 6.121796588s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:42:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.12182336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.776600699s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:42:52 UTC) orbiter warning: unexpected EOF Duration: 5.77663845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.427485922s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:42:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427508785s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)