Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.691245683s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:45:19 UTC) orbiter warning: unexpected EOF Duration: 5.691278687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 4.325568182s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:45:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.32560404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 5.780191681s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.780247597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.751712767s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:44:28 UTC) orbiter warning: unexpected EOF Duration: 5.751805133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.44953561s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:44:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449565277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.636010789s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:43:59 UTC) orbiter warning: unexpected EOF Duration: 5.636043321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.425671873s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:43:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425697551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.838856413s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:43:24 UTC) orbiter warning: unexpected EOF Duration: 5.838888635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 5.897304572s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:43:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.897337416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 6.121796588s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:42:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.12182336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.776600699s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:42:52 UTC) orbiter warning: unexpected EOF Duration: 5.77663845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.427485922s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:42:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427508785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 5.956309333s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:42:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.956341626s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.825163965s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:42:16 UTC) orbiter warning: unexpected EOF Duration: 5.825195235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.590949583s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:41:50 UTC) orbiter warning: unexpected EOF Duration: 5.590976143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.386553458s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:41:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386581161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 5.974476069s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:41:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.974503301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.731073494s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:41:07 UTC) orbiter warning: unexpected EOF Duration: 5.731100335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.441469469s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:40:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4415021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.358767907s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:40:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.35879547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 5.803787099s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:40:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.803817416s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.66684934s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:40:23 UTC) orbiter warning: unexpected EOF Duration: 5.666877953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.634195129s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:39:45 UTC) orbiter warning: unexpected EOF Duration: 5.634224507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 5.855670295s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:39:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.855696616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.521799365s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:39:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521828429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.86373816s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:39:10 UTC) orbiter warning: unexpected EOF Duration: 5.863767697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.367450671s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:38:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367483243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 5.962318448s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:38:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.962343515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.658923648s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:38:03 UTC) orbiter warning: unexpected EOF Duration: 5.658962031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.41170535s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:37:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411722642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 5.949010825s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:37:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.949040651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.423155505s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:37:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423189829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.715130728s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:37:16 UTC) orbiter warning: unexpected EOF Duration: 5.715156006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.678669962s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:36:58 UTC) orbiter warning: unexpected EOF Duration: 5.678713454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 6.081944979s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.081977691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.352434945s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:36:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.352466305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.484562916s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:36:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484592073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.671513833s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:36:01 UTC) orbiter warning: unexpected EOF Duration: 5.671541975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 5.929389743s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:35:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.92941921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.630091215s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:35:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.63011028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.846880973s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:35:08 UTC) orbiter warning: unexpected EOF Duration: 5.846995521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.787206629s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:34:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.787234643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 5.86048476s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:34:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.860514427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 5.997369881s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:34:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.997388387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.785055156s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:33:57 UTC) orbiter warning: unexpected EOF Duration: 5.785119528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.498226212s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:33:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498256861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.662388735s trace: https://jaeger.dev.storj.io/trace/6be364084c584eb4 FAIL (2025-11-13 00:33:31 UTC) orbiter warning: unexpected EOF Duration: 5.662492583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.421235548s trace: https://jaeger.dev.storj.io/trace/6b0d655184bb6103 FAIL (2025-11-13 00:33:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421287016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 5.889446222s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:33:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.889469397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 5.872989687s trace: https://jaeger.dev.storj.io/trace/6aa382433dec8ce1 FAIL (2025-11-13 00:32:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 277 (88h21m38.842598748s) [upload=77786]: Starting upload Nov 13 00:25:40.291219981 (88h21m38.842690462s) [upload=77786]: Done starting upload Nov 13 00:25:40.291350610 (88h21m38.842821091s) [upload=77786]: Uploading segments... Nov 13 00:25:40.291488783 (88h21m38.842959254s) [upload=77786]: Got next segment. Inline: false Nov 13 00:25:40.291534580 (88h21m38.843005061s) [upload=77786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 00:25:40.291741053 (88h21m38.843211514s) [upload=77786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 00:25:40.545722199 (88h21m39.09719266s) [seg_pos={0 0}, upload=77786]: Begin upload segment... Nov 13 00:25:40.545777725 (88h21m39.097248186s) [seg_pos={0 0}, upload=77786]: Done begin upload segment. Nov 13 00:25:40.545892393 (88h21m39.097362855s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547764438 (88h21m39.099234889s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.547895368 (88h21m39.099365829s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.549124428 (88h21m39.100594889s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550058066 (88h21m39.101528537s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.550858470 (88h21m39.102328921s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.551776608 (88h21m39.103247069s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552469598 (88h21m39.103940059s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552560201 (88h21m39.104030662s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.552991291 (88h21m39.104461752s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.553426480 (88h21m39.104896941s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.555827743 (88h21m39.107298194s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556313768 (88h21m39.107784219s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.556761381 (88h21m39.108231832s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.558179470 (88h21m39.109649961s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:40.559023518 (88h21m39.110493979s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Uploading piece... Nov 13 00:25:42.281920169 (88h21m40.833390631s) [upload=77786]: Next returned nil segment Nov 13 00:25:42.281938253 (88h21m40.833408714s) [upload=77786]: Waiting for error group managing segments... Nov 13 00:25:42.618108135 (88h21m41.169578596s) [noise=true , address=77.90.31.11:4554 , piece_id=4QZDKLWILRYIYYFPLHN2WPKEC4YYB4BLNU6NWZMN4DUQUBLBEERA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.682422013 (88h21m41.233892474s) [noise=true , address=45.91.116.6:4975 , piece_id=EDCMSHKNZLMW2SGVSSSA7P7K65YISCY3NZJ7DZHYEATZMXPDEKVA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.687278060 (88h21m41.238748521s) [noise=true , address=77.90.31.19:4843 , piece_id=MI4KQZHGEX3JH6HD5VBEQRUMR6XVEJTXJJ47MPKY7NXZ6YRMJKKQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.707278465 (88h21m41.258748926s) [noise=true , address=45.91.116.10:4081 , piece_id=UIXPJIPK4KSVSWI3AQWQ6CQN6V2X7L4UZMGTLTJHBLDKL5DK3NNA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:42.742723731 (88h21m41.294194192s) [noise=true , address=77.90.31.21:4878 , piece_id=JVBEIEA5RAV3777TGNL7RM6MCNLFDRR5AJBBIPBECFAEN5MNT3ZA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:58.871748192 (88h21m57.423218653s) [noise=true , address=145.79.227.18:4006 , piece_id=K4X7C4JB6VXQ634OA5RVE4CW2TDTL3ZRJVMNLX4UGAMPPL4XJD3Q, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.670592207 (88h21m58.222062668s) [noise=true , address=145.79.227.27:4690 , piece_id=SBNB5GZUF3QEKC24B7HYD2EQYKPFRECDISEVNXDO4NRNRHGVOB4A, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:25:59.886822126 (88h21m58.438292587s) [noise=true , address=145.79.227.7:4294 , piece_id=ADLTT37QAHGGGKGGYMSKYHNORQFJLZGFZOAC7DKGRRTWNDRYVMPA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:02.440128983 (88h22m0.991599434s) [noise=true , address=145.79.227.28:4725 , piece_id=DDTJ36VP7ZZZHL2XQJEZDQ2UFREQSUH4P4QILHKNMAXV74ZI2JXA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: <nil> Nov 13 00:26:09.584279709 (88h22m8.135750171s) [noise=true , address=145.79.227.9:4366 , piece_id=NX46BR5S5JAIQJSXSVLISUTAHM4XAGKV2BC4TBO44CXT36NWWCOA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56764->145.79.227.9:4366: use of closed network connection Nov 13 00:26:09.584518174 (88h22m8.135988655s) [noise=true , address=145.79.227.11:4732 , piece_id=NMEEYPORS6LMD2KNNCSUH6IL6Y37SD6BMOUUF6QFEP6XSYK7U2BA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.584641228 (88h22m8.13611169s) [noise=true , address=145.79.227.17:4971 , piece_id=PKAHMU2PZDWKIOGWH4XUK6RNSQCAJNIMRXX7KWDRFVOCYXUAAPSQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53052->145.79.227.17:4971: use of closed network connection Nov 13 00:26:09.585215862 (88h22m8.136686323s) [noise=true , address=145.79.227.6:4257 , piece_id=EI45Q6ZVDDE6EIPXJG2UXN7X7CRU56L7SFIURGRYRB64UAEX3OLA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585386968 (88h22m8.136857429s) [noise=true , address=145.79.227.15:4873 , piece_id=ABKC4IWRD7THYMNW7HDQUBZIOCYJSGDALPUZDZSEW26OL7WJX6DQ, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.585687090 (88h22m8.137157551s) [noise=true , address=145.79.227.30:4761 , piece_id=QOG57DUJK74LNGWJ5ICAZNL3HET7KQV56Z66CHYNE2PMU6HIWYUA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38800->145.79.227.30:4761: use of closed network connection Nov 13 00:26:09.586213542 (88h22m8.137684003s) [noise=true , address=145.79.227.25:4619 , piece_id=GTVZKATL73ESZRC6EBBEU7ZUBRKUVNHNPHLDQB25S6TA4DHTBEAA, seg_pos={0 0}, upload=77786]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 00:26:09.586324123 (88h22m8.137794584s) [seg_pos={0 0}, upload=77786]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 00:26:09.834191416 (88h22m8.385661897s) [upload=77786]: Done uploading segments... Nov 13 00:26:09.834234117 (88h22m8.385704588s) [upload=77786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.873086181s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)