Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XX..XX.XX..XXX............XXXX.....XXX..X.....X......] duration: 29.391002408s FAIL (2025-07-26 23:43:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.391026935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.942575029s FAIL (2025-07-26 23:42:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.942607591s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X..] duration: 7.533406287s FAIL (2025-07-26 23:42:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.533426455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [....X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235934942s FAIL (2025-07-26 23:42:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 6 23:42:17.279742706 (60h6m27.002305747s) [upload=57668]: Done starting upload Jul 26 23:42:17.279847177 (60h6m27.002410277s) [upload=57668]: Uploading segments... Jul 26 23:42:17.279974871 (60h6m27.002537921s) [upload=57668]: Got next segment. Inline: false Jul 26 23:42:17.280026389 (60h6m27.002589429s) [upload=57668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:17.280035156 (60h6m27.002598206s) [upload=57668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:17.512742705 (60h6m27.235305766s) [seg_pos={0 0}, upload=57668]: Begin upload segment... Jul 26 23:42:17.512804884 (60h6m27.235367924s) [seg_pos={0 0}, upload=57668]: Done begin upload segment. Jul 26 23:42:17.512826726 (60h6m27.235389766s) [noise=true , address=104.36.82.16:4294 , piece_id=HZXONEUORUGVCUB7XNRCFK5XKYRQAUUXN3GQ64LEDCZHDIZAFIDQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.512894315 (60h6m27.235457355s) [noise=true , address=45.91.116.13:4155 , piece_id=4YCZVXPO6TP2XHXEF3YLXA4DVJ5PSCDPVLMV3VY7R3ZCOIXMHZNQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513163961 (60h6m27.235727001s) [noise=true , address=77.90.31.18:4805 , piece_id=XDKJXPHMWK7XAXANBIPYPUC6C22NQ2NIZVFR2Y4TG7K42LAX3AIQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513377560 (60h6m27.23594059s) [noise=true , address=104.36.82.4:4803 , piece_id=F7HYVQICYIYICNHJL3SOK3BIHDZQ3PM3AUB4CUX3KXEO5RQZN3ZQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513410964 (60h6m27.235973994s) [noise=true , address=104.36.82.11:4115 , piece_id=SIPBBGCWRCQHBO4PEBW4F5TTHT75HKMLCZKAYMUOQTWK3JLHJRSQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513708623 (60h6m27.236271663s) [noise=true , address=104.36.82.14:4222 , piece_id=GDBCOOJGDW3VQJUNHAROMA2SLWENNETTGOWUBUYSN4ZWBPB6SU4A, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513753499 (60h6m27.236316529s) [noise=true , address=45.91.116.6:4975 , piece_id=GCHZT5PYMPMR3FZUWRSJEJSLYW63TRB63PLQHKJGFORHFMAR5NHA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514013717 (60h6m27.236576757s) [noise=true , address=104.36.82.13:4185 , piece_id=LLHIVKYSITYLRLW5A5FC75WEPA2M4WBVMV5B7MXYETGUOA6UICTA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514089142 (60h6m27.236652182s) [noise=true , address=104.36.82.2:4732 , piece_id=RGFMOCNUXXA7VCA3HJ3PJOQNZIS6KTEU7GF6F43PKHWXO2W4G5NA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514230993 (60h6m27.236794033s) [noise=true , address=104.36.82.3:4765 , piece_id=5JZKCPZFZLDY4SRH7NHCL7ML3M4UMLVMD5BHDDS5EWEDTQWV6GLA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514422880 (60h6m27.23698592s) [noise=true , address=45.91.116.7:4010 , piece_id=V4UYQF2M5N6SSZXIVEHTDWGLCUUTBHJXUSE46F6LPUQJVZXZI3PA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514453619 (60h6m27.237016659s) [noise=true , address=77.90.31.11:4553 , piece_id=UJ3ZHSDZMEEJOIBOMCZ6BOJ2RY5KDH6CCINSUE6CJIOPVKRJGDKA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514670043 (60h6m27.237233083s) [noise=true , address=77.90.31.21:4877 , piece_id=QUYBJ6CPBEVQMZVXVEIVZSFGHIFMY6AZLNK3FGVJGN4AJ6OEBMFQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514780314 (60h6m27.237343354s) [noise=true , address=45.91.116.9:4261 , piece_id=LH4ICVQDJ2LURIATIFFLVPCDDYKBICI2WJODDHFOUXH2EA6OUDAQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.515038177 (60h6m27.237601207s) [noise=true , address=77.90.31.14:4663 , piece_id=VVNLCJV5QVQBRG47IVEBPRMVVQRBG4MELSDUYQGFYIHQTTDMP3XQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.517980547 (60h6m27.240543587s) [noise=true , address=104.36.82.15:4259 , piece_id=BREAEDTOOGFUDI7MFELAU5LFZFSRFNSGAIDSV3S6BAVT2CARVHSA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:19.269360754 (60h6m28.991923794s) [upload=57668]: Next returned nil segment Jul 26 23:42:19.269380342 (60h6m28.991943382s) [upload=57668]: Waiting for error group managing segments... Jul 26 23:42:19.545421040 (60h6m29.26798408s) [noise=true , address=45.91.116.13:4155 , piece_id=4YCZVXPO6TP2XHXEF3YLXA4DVJ5PSCDPVLMV3VY7R3ZCOIXMHZNQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.624565276 (60h6m29.347128316s) [noise=true , address=45.91.116.6:4975 , piece_id=GCHZT5PYMPMR3FZUWRSJEJSLYW63TRB63PLQHKJGFORHFMAR5NHA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.626840720 (60h6m29.34940376s) [noise=true , address=77.90.31.11:4553 , piece_id=UJ3ZHSDZMEEJOIBOMCZ6BOJ2RY5KDH6CCINSUE6CJIOPVKRJGDKA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.626891127 (60h6m29.349454157s) [noise=true , address=45.91.116.7:4010 , piece_id=V4UYQF2M5N6SSZXIVEHTDWGLCUUTBHJXUSE46F6LPUQJVZXZI3PA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.632696274 (60h6m29.355259324s) [noise=true , address=77.90.31.14:4663 , piece_id=VVNLCJV5QVQBRG47IVEBPRMVVQRBG4MELSDUYQGFYIHQTTDMP3XQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.852411615 (60h6m29.574974655s) [noise=true , address=45.91.116.9:4261 , piece_id=LH4ICVQDJ2LURIATIFFLVPCDDYKBICI2WJODDHFOUXH2EA6OUDAQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:21.206148120 (60h6m30.92871115s) [noise=true , address=77.90.31.18:4805 , piece_id=XDKJXPHMWK7XAXANBIPYPUC6C22NQ2NIZVFR2Y4TG7K42LAX3AIQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:46.640897359 (60h6m56.363460399s) [noise=true , address=104.36.82.14:4222 , piece_id=GDBCOOJGDW3VQJUNHAROMA2SLWENNETTGOWUBUYSN4ZWBPB6SU4A, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:46.640965670 (60h6m56.36352871s) [noise=true , address=104.36.82.11:4115 , piece_id=SIPBBGCWRCQHBO4PEBW4F5TTHT75HKMLCZKAYMUOQTWK3JLHJRSQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.640972293 (60h6m56.363535333s) [noise=true , address=104.36.82.4:4803 , piece_id=F7HYVQICYIYICNHJL3SOK3BIHDZQ3PM3AUB4CUX3KXEO5RQZN3ZQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:46.641009625 (60h6m56.363572665s) [noise=true , address=104.36.82.2:4732 , piece_id=RGFMOCNUXXA7VCA3HJ3PJOQNZIS6KTEU7GF6F43PKHWXO2W4G5NA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:35264->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641022629 (60h6m56.363585689s) [noise=true , address=104.36.82.16:4294 , piece_id=HZXONEUORUGVCUB7XNRCFK5XKYRQAUUXN3GQ64LEDCZHDIZAFIDQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:42:46.641041676 (60h6m56.363604716s) [noise=true , address=104.36.82.15:4259 , piece_id=BREAEDTOOGFUDI7MFELAU5LFZFSRFNSGAIDSV3S6BAVT2CARVHSA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:60152->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641093846 (60h6m56.363656886s) [noise=true , address=77.90.31.21:4877 , piece_id=QUYBJ6CPBEVQMZVXVEIVZSFGHIFMY6AZLNK3FGVJGN4AJ6OEBMFQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641111339 (60h6m56.363674379s) [noise=true , address=104.36.82.13:4185 , piece_id=LLHIVKYSITYLRLW5A5FC75WEPA2M4WBVMV5B7MXYETGUOA6UICTA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641124014 (60h6m56.363687054s) [noise=true , address=104.36.82.3:4765 , piece_id=5JZKCPZFZLDY4SRH7NHCL7ML3M4UMLVMD5BHDDS5EWEDTQWV6GLA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641135906 (60h6m56.363698946s) [seg_pos={0 0}, upload=57668]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:42:46.875081343 (60h6m56.597644424s) [upload=57668]: Done uploading segments... Jul 26 23:42:46.875111722 (60h6m56.597674763s) [upload=57668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.235959067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237679062s FAIL (2025-07-26 23:42:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.237701834s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.833028084s FAIL (2025-07-26 23:42:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833053382s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X.....] duration: 27.667436505s FAIL (2025-07-26 23:42:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.667452515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.249836979s FAIL (2025-07-26 23:42:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== m12.532702764s) [seg_pos={0 0}, upload=57647]: Done begin upload segment. Jul 26 23:41:02.810161366 (60h5m12.532724406s) [noise=true , address=79.127.201.215:4350 , piece_id=HINDTR3MT4SHVNU4Y6EVA5BVA6BRZX3ILCLBYFWOVA62GGATU3NQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810277728 (60h5m12.532840768s) [noise=true , address=79.127.205.246:4275 , piece_id=E73QCW5QAG7ZZSDSZEKANVXIN2CIDIJBX556XHY3DWX6K7CF3TFQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810545791 (60h5m12.533108831s) [noise=true , address=79.127.220.97:4361 , piece_id=MHF6V6WHTLX6MG2SIZTMSAP6RA4A4ZRDKVHTS727Z5GCQM7G242Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810781161 (60h5m12.533344201s) [noise=true , address=104.36.82.53:4156 , piece_id=5F6UPSHANOQWQNL7SYHWFWBU7H7OTF2EZ2EHMOLVLRQKAW6VOMHQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810806700 (60h5m12.53336973s) [noise=true , address=79.127.205.235:4685 , piece_id=JYAJ2NLIVVE4ILXE5YG7TM7ZQ43R44FJUHRSSCXVKCC2KLE3ANMA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811054224 (60h5m12.533617264s) [noise=true , address=104.36.82.42:4821 , piece_id=HLVQVVUPPELZTNMDZKAGYYFTJA7TSWO5BVV4EQBZP7B5SQWVHJ3Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811150719 (60h5m12.533713759s) [noise=true , address=64.49.13.140:4508 , piece_id=GDY3EEGJB2CIVBL3LZ5EDKDQ7KCC5Q3LJWBFABPNGS6ZH4NZ7LDA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811277701 (60h5m12.533840731s) [noise=true , address=104.36.82.52:4114 , piece_id=MOOXBMSDUSMZPAOSFCBUF6LHB52JT2S3H3VKMFTGYSIXSFIIBVCA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811453749 (60h5m12.534016789s) [noise=true , address=77.90.31.21:4895 , piece_id=PY4S7H5WDUBPFCDINMNNGWQV2RCLXEAAYKKEGVH3F7PLU6K666TA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811516639 (60h5m12.534079679s) [noise=true , address=104.36.82.56:4249 , piece_id=UZ76WXHIZQMVQ6VZW7VMS2AJUZPGGKMIZWXCOWGURD37PZKOWDDA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811744524 (60h5m12.534307564s) [noise=true , address=79.127.201.214:4321 , piece_id=2LBSADAWPVHIXH5KYBYX7NXQXLFUKGY4GJEM5EDXH6AQBPADIBNQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811756237 (60h5m12.534319267s) [noise=true , address=77.90.31.15:4714 , piece_id=3YFQ6PUG5H5LH6BNP4E44GJTMUJVIGO4OOYFE3VAGFMN6GODBP6Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811978362 (60h5m12.534541402s) [noise=true , address=79.127.205.234:4607 , piece_id=W7UWXV6SWISPTD3OELN4UDWFIQR44T2BAQWTXRAFVZ3BYWINSMBA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812056752 (60h5m12.534619792s) [noise=true , address=104.36.82.44:4875 , piece_id=SYKIQDVGSDG7CZ5CTHWYXD24MPTXJWPVYK2Z7VWF5DI4BSF3FIJQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812269640 (60h5m12.53483268s) [noise=true , address=104.36.82.47:4531 , piece_id=BLQISDEF5BOHTR43CS7IOLRIMEUQSAVQMQOZG7S6IF2XBF646TSA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812404498 (60h5m12.534967538s) [noise=true , address=79.127.219.47:4048 , piece_id=VVETUC72MUYLONSHR5BF4FBFVR7NAW66KPIPYME2WD75QJ7WFQ2A, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812674213 (60h5m12.535237253s) [noise=true , address=77.90.31.12:4597 , piece_id=YA5XSKX4U3JJGAI7NIAU2MEYFV2GCNHDH6E3EVWRGLZIMWRQYJDQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.814084693 (60h5m12.536647723s) [noise=true , address=79.127.220.101:4041 , piece_id=UKGLEVF5VSYK6IQM52MTEDRHIPHNJDBA2VKD7MN7NRZXCES34MZQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:03.830228169 (60h5m13.552791219s) [noise=true , address=79.127.220.97:4361 , piece_id=MHF6V6WHTLX6MG2SIZTMSAP6RA4A4ZRDKVHTS727Z5GCQM7G242Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 23:41:03.919165754 (60h5m13.641728784s) [upload=57647]: Next returned nil segment Jul 26 23:41:03.919181674 (60h5m13.641744704s) [upload=57647]: Waiting for error group managing segments... Jul 26 23:41:04.199928297 (60h5m13.922491337s) [noise=true , address=79.127.205.234:4607 , piece_id=W7UWXV6SWISPTD3OELN4UDWFIQR44T2BAQWTXRAFVZ3BYWINSMBA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.208241703 (60h5m13.930804743s) [noise=true , address=79.127.205.235:4685 , piece_id=JYAJ2NLIVVE4ILXE5YG7TM7ZQ43R44FJUHRSSCXVKCC2KLE3ANMA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.227137626 (60h5m13.949700666s) [noise=true , address=79.127.205.246:4275 , piece_id=E73QCW5QAG7ZZSDSZEKANVXIN2CIDIJBX556XHY3DWX6K7CF3TFQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.681758362 (60h5m14.404321402s) [noise=true , address=79.127.219.47:4048 , piece_id=VVETUC72MUYLONSHR5BF4FBFVR7NAW66KPIPYME2WD75QJ7WFQ2A, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.747406818 (60h5m14.469969868s) [noise=true , address=77.90.31.12:4597 , piece_id=YA5XSKX4U3JJGAI7NIAU2MEYFV2GCNHDH6E3EVWRGLZIMWRQYJDQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.844545002 (60h5m14.567108062s) [noise=true , address=77.90.31.15:4714 , piece_id=3YFQ6PUG5H5LH6BNP4E44GJTMUJVIGO4OOYFE3VAGFMN6GODBP6Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:05.652683113 (60h5m15.375246153s) [noise=true , address=79.127.220.101:4041 , piece_id=UKGLEVF5VSYK6IQM52MTEDRHIPHNJDBA2VKD7MN7NRZXCES34MZQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:07.253427507 (60h5m16.975990547s) [noise=true , address=79.127.201.214:4321 , piece_id=2LBSADAWPVHIXH5KYBYX7NXQXLFUKGY4GJEM5EDXH6AQBPADIBNQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:07.546097763 (60h5m17.268660803s) [noise=true , address=64.49.13.140:4508 , piece_id=GDY3EEGJB2CIVBL3LZ5EDKDQ7KCC5Q3LJWBFABPNGS6ZH4NZ7LDA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:27.601213531 (60h5m37.323776591s) [noise=true , address=79.127.201.215:4350 , piece_id=HINDTR3MT4SHVNU4Y6EVA5BVA6BRZX3ILCLBYFWOVA62GGATU3NQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:42:01.706639956 (60h6m11.429202996s) [noise=true , address=104.36.82.52:4114 , piece_id=MOOXBMSDUSMZPAOSFCBUF6LHB52JT2S3H3VKMFTGYSIXSFIIBVCA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:01.706679952 (60h6m11.429242992s) [noise=true , address=104.36.82.56:4249 , piece_id=UZ76WXHIZQMVQ6VZW7VMS2AJUZPGGKMIZWXCOWGURD37PZKOWDDA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 188.166.225.170:56940->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706721212 (60h6m11.429284252s) [noise=true , address=77.90.31.21:4895 , piece_id=PY4S7H5WDUBPFCDINMNNGWQV2RCLXEAAYKKEGVH3F7PLU6K666TA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: read tcp 188.166.225.170:41850->77.90.31.21:4895: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706767570 (60h6m11.42933061s) [noise=true , address=104.36.82.42:4821 , piece_id=HLVQVVUPPELZTNMDZKAGYYFTJA7TSWO5BVV4EQBZP7B5SQWVHJ3Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706792327 (60h6m11.429355367s) [noise=true , address=104.36.82.47:4531 , piece_id=BLQISDEF5BOHTR43CS7IOLRIMEUQSAVQMQOZG7S6IF2XBF646TSA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706810943 (60h6m11.429373983s) [noise=true , address=104.36.82.53:4156 , piece_id=5F6UPSHANOQWQNL7SYHWFWBU7H7OTF2EZ2EHMOLVLRQKAW6VOMHQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: rpc: dial tcp 104.36.82.53:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706881799 (60h6m11.429444839s) [noise=true , address=104.36.82.44:4875 , piece_id=SYKIQDVGSDG7CZ5CTHWYXD24MPTXJWPVYK2Z7VWF5DI4BSF3FIJQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:104.36.82.44:4875): protocol: rpc: dial tcp 104.36.82.44:4875: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:42:01.706898761 (60h6m11.429461801s) [seg_pos={0 0}, upload=57647]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:42:01.954524682 (60h6m11.677087722s) [upload=57647]: Done uploading segments... Jul 26 23:42:01.954556963 (60h6m11.677120013s) [upload=57647]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.249850715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [.....X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.238261725s FAIL (2025-07-26 23:41:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== stItem_SegmentBegin Jul 26 23:41:24.496328373 (60h5m34.218891433s) [seg_pos={0 0}, upload=57654]: Begin upload segment... Jul 26 23:41:24.496381976 (60h5m34.218945016s) [seg_pos={0 0}, upload=57654]: Done begin upload segment. Jul 26 23:41:24.496413005 (60h5m34.218976075s) [noise=true , address=104.36.82.5:4840 , piece_id=PQEMYHQZFRYZJH7PGVQNFUSBFVRPFJQ4ED3SEJMQZGL3DACIHOWA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.496449575 (60h5m34.219012615s) [noise=true , address=45.91.116.8:4048 , piece_id=BJEJM22SFWOXJ7Z34UJATNP3EIUIGRR6RQL2EXRAHEXMIENO2FLQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.496930134 (60h5m34.219493174s) [noise=true , address=104.36.82.5:4837 , piece_id=BA5NDHGCADIZJWK2I2U5TEEOLRZP7SSNUOMPC2KV7YUNI3KM4DFA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497029315 (60h5m34.219592365s) [noise=true , address=104.36.82.14:4223 , piece_id=ZOTC3VW3RSTPR5IQGSDHD2PFOMEI3ZLSPB5AJUO7E3MJNM4EBHGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497339177 (60h5m34.219902217s) [noise=true , address=104.36.82.13:4186 , piece_id=PGQCAIPOUICCDLRB4VXOXPRZO3YL5HOJLRVL7TCUAWHPZDMVCGGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497736147 (60h5m34.220299197s) [noise=true , address=45.91.116.10:4083 , piece_id=XO3G5WPRNPIJ6IYYP7R73TGJSILGURNGFUEZ66K56RXZWD7WYPZA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498120503 (60h5m34.220683553s) [noise=true , address=104.36.82.6:4875 , piece_id=QD7QKQEFENY3MR754BBBOKPYKGPJA6F3UHND5KGMJO6D7JQTCZEQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498451135 (60h5m34.221014175s) [noise=true , address=104.36.82.9:4005 , piece_id=AK7VG6ZHPQGMA2X3IUGU46F7GXLL57FGNHMRN2HP24K4UKGJNTGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498751699 (60h5m34.221314739s) [noise=true , address=104.36.82.8:4971 , piece_id=MXFGKXD5IYSV3JVL5A7WHINHYTRR7EEHHI7VLSN3ZMW5HVUOPUUA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499076571 (60h5m34.221639611s) [noise=true , address=45.91.116.7:4009 , piece_id=ULX6U7BOCMBCUOMYW7JA37BGX7AKA3FYTLAV4Z65SXRV7Y5ZXAYQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499352048 (60h5m34.221915088s) [noise=true , address=104.36.82.11:4115 , piece_id=BCIRCPFTDOSPSPA7ELRYX47IKZ6OZMRGV2FTXWIRBEKPGDQ2IUGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499666310 (60h5m34.22222935s) [noise=true , address=104.36.82.9:4007 , piece_id=ZVHQ5WMYOJWG7CI3I6TF2O65PARCRY5KJ2WJTXQHXGPM53UEEPBA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499973137 (60h5m34.222536177s) [noise=true , address=45.91.116.4:4937 , piece_id=RWLIKN33SREMPPV2SREPCFQZO57N3OM7IUUCVGI2YCF754QFTUAA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500298129 (60h5m34.222861169s) [noise=true , address=45.91.116.6:4974 , piece_id=WADUCHLS6HELMRISN76TN4X4WJKRRLUTTVS4T7BJVUJM3Y3F3CBA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500594335 (60h5m34.223157375s) [noise=true , address=104.36.82.13:4187 , piece_id=FZ7SV2V2ZL3YZ2ADXAJ65L5SVHNV7DOFWKLIGHC5W3H76W45DBIA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500904559 (60h5m34.223467599s) [noise=true , address=45.91.116.12:4225 , piece_id=LOV5ZIIUNRXYZ5GXFAQTTOORZHO7IL7W4KJ5SNEGSRLO535CPDFQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:26.259926216 (60h5m35.982489246s) [upload=57654]: Next returned nil segment Jul 26 23:41:26.259943168 (60h5m35.982506208s) [upload=57654]: Waiting for error group managing segments... Jul 26 23:41:26.535548654 (60h5m36.258111694s) [noise=true , address=45.91.116.7:4009 , piece_id=ULX6U7BOCMBCUOMYW7JA37BGX7AKA3FYTLAV4Z65SXRV7Y5ZXAYQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.591695871 (60h5m36.314258911s) [noise=true , address=45.91.116.4:4937 , piece_id=RWLIKN33SREMPPV2SREPCFQZO57N3OM7IUUCVGI2YCF754QFTUAA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.620833340 (60h5m36.34339637s) [noise=true , address=45.91.116.12:4225 , piece_id=LOV5ZIIUNRXYZ5GXFAQTTOORZHO7IL7W4KJ5SNEGSRLO535CPDFQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.621147070 (60h5m36.34371011s) [noise=true , address=45.91.116.6:4974 , piece_id=WADUCHLS6HELMRISN76TN4X4WJKRRLUTTVS4T7BJVUJM3Y3F3CBA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.628699220 (60h5m36.35126226s) [noise=true , address=45.91.116.8:4048 , piece_id=BJEJM22SFWOXJ7Z34UJATNP3EIUIGRR6RQL2EXRAHEXMIENO2FLQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.765689543 (60h5m36.488252583s) [noise=true , address=45.91.116.10:4083 , piece_id=XO3G5WPRNPIJ6IYYP7R73TGJSILGURNGFUEZ66K56RXZWD7WYPZA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:53.619874012 (60h6m3.342437052s) [noise=true , address=104.36.82.14:4223 , piece_id=ZOTC3VW3RSTPR5IQGSDHD2PFOMEI3ZLSPB5AJUO7E3MJNM4EBHGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:36664->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619917936 (60h6m3.342480976s) [noise=true , address=104.36.82.9:4005 , piece_id=AK7VG6ZHPQGMA2X3IUGU46F7GXLL57FGNHMRN2HP24K4UKGJNTGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619947413 (60h6m3.342510453s) [noise=true , address=104.36.82.8:4971 , piece_id=MXFGKXD5IYSV3JVL5A7WHINHYTRR7EEHHI7VLSN3ZMW5HVUOPUUA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:42816->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619975226 (60h6m3.342538266s) [noise=true , address=104.36.82.13:4187 , piece_id=FZ7SV2V2ZL3YZ2ADXAJ65L5SVHNV7DOFWKLIGHC5W3H76W45DBIA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620005053 (60h6m3.342568093s) [noise=true , address=104.36.82.5:4837 , piece_id=BA5NDHGCADIZJWK2I2U5TEEOLRZP7SSNUOMPC2KV7YUNI3KM4DFA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:46530->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620032275 (60h6m3.342595305s) [noise=true , address=104.36.82.6:4875 , piece_id=QD7QKQEFENY3MR754BBBOKPYKGPJA6F3UHND5KGMJO6D7JQTCZEQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:48892->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620059908 (60h6m3.342622948s) [noise=true , address=104.36.82.11:4115 , piece_id=BCIRCPFTDOSPSPA7ELRYX47IKZ6OZMRGV2FTXWIRBEKPGDQ2IUGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620086899 (60h6m3.34264994s) [noise=true , address=104.36.82.13:4186 , piece_id=PGQCAIPOUICCDLRB4VXOXPRZO3YL5HOJLRVL7TCUAWHPZDMVCGGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:39234->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620114392 (60h6m3.342677422s) [noise=true , address=104.36.82.9:4007 , piece_id=ZVHQ5WMYOJWG7CI3I6TF2O65PARCRY5KJ2WJTXQHXGPM53UEEPBA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620143828 (60h6m3.342706868s) [noise=true , address=104.36.82.5:4840 , piece_id=PQEMYHQZFRYZJH7PGVQNFUSBFVRPFJQ4ED3SEJMQZGL3DACIHOWA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:32910->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620359511 (60h6m3.342922551s) [seg_pos={0 0}, upload=57654]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:41:53.856061540 (60h6m3.57862458s) [upload=57654]: Done uploading segments... Jul 26 23:41:53.856086668 (60h6m3.578649708s) [upload=57654]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.238276393s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X.] duration: 7.70407651s FAIL (2025-07-26 23:41:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.704102448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [......X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.233307354s FAIL (2025-07-26 23:41:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== l 26 23:40:53.798921398 (60h5m3.521484438s) [upload=57643]: Starting upload Jul 26 23:40:53.798984470 (60h5m3.52154751s) [upload=57643]: Done starting upload Jul 26 23:40:53.799053832 (60h5m3.521616872s) [upload=57643]: Uploading segments... Jul 26 23:40:53.799162290 (60h5m3.52172533s) [upload=57643]: Got next segment. Inline: false Jul 26 23:40:53.799196314 (60h5m3.521759354s) [upload=57643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:40:53.799209410 (60h5m3.52177245s) [upload=57643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:40:54.065144082 (60h5m3.787707132s) [seg_pos={0 0}, upload=57643]: Begin upload segment... Jul 26 23:40:54.065204337 (60h5m3.787767377s) [seg_pos={0 0}, upload=57643]: Done begin upload segment. Jul 26 23:40:54.065233834 (60h5m3.787796874s) [noise=true , address=45.91.116.10:4083 , piece_id=XIPEAMLQ7XASULSDU4IYGFZODUMXKWC4HIBDTSF57ECREBALVMYA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065288709 (60h5m3.787851749s) [noise=true , address=104.36.82.13:4187 , piece_id=G3VQ6UIIWQXTPFZYFGQ25HWODTBAQZYWVVAJWOLOS5ALNJGDAHBA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065838520 (60h5m3.788401581s) [noise=true , address=45.91.116.12:4225 , piece_id=W3AJYCVJZ5X7O3M3ERKIUJRVBKQBZ5JPBQI2OQP2ZUDBGDMGKP3Q, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065924976 (60h5m3.788488016s) [noise=true , address=45.91.116.6:4973 , piece_id=HSQIBXKQNP3LTRF6SPEYXJVUHLT3Q3EB6XSOPA2AQ7N3SID73AFA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066242133 (60h5m3.788805173s) [noise=true , address=104.36.82.5:4839 , piece_id=4NVOW5SCB6357X4OC2LBN5UDKN73JD3RACG6KTFG7LUC3ECD5MEQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066343667 (60h5m3.788906707s) [noise=true , address=104.36.82.9:4007 , piece_id=P4ZZAX4H7SNII47EWOF4JZCZ2CCTGXJFXHVBFXYFF5Q6KXHRR6WQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066682385 (60h5m3.789245425s) [noise=true , address=45.91.116.11:4118 , piece_id=7QQIPZSUOTQ6SULT4UP2CT5JXZE2DGTOWKCEJ54KLYD573GPFDSQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066779541 (60h5m3.789342581s) [noise=true , address=104.36.82.8:4969 , piece_id=YEXVSD6POLNBCC6QWSBVAT2CBCPPRAUOBTOT2MJXQQ3SNFTBTIJQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067030912 (60h5m3.789593952s) [noise=true , address=104.36.82.4:4802 , piece_id=SRNVGYMPYK7BJBX2MZGZ3CRTZ6O4TCY6TTHDSUUIXZHCNLOBLMFQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067184335 (60h5m3.789747376s) [noise=true , address=104.36.82.16:4293 , piece_id=VXLOQOHR7ZOX6QV4K252TARFU7ERB7WSTLO6PQR3LVNK3PWOSNSQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067375692 (60h5m3.789938732s) [noise=true , address=104.36.82.7:4912 , piece_id=C3NV4RSZGQZYX7WOBXL3QQJPFNLGVYFQVYBOAU2ZWE5EBZGXIF5A, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067563241 (60h5m3.790126281s) [noise=true , address=104.36.82.10:4078 , piece_id=5KZ3MYPM3QI6DEHKSFRVCMHBV3XLTBRDSPSTBHOFUTCJFHVTLWZQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067733887 (60h5m3.790296927s) [noise=true , address=45.91.116.8:4045 , piece_id=K6EU6IWY2C4JNFKE3GEHO2TXITUR4NDR36YNQ4DT574N3TDVX2YA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067955040 (60h5m3.79051808s) [noise=true , address=45.91.116.4:4938 , piece_id=ZBHBICLFFLQZCUU5M6LPN7RBQR2ABGVC6XWID3ZNXF5KTJ22LBDQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.068047907 (60h5m3.790610947s) [noise=true , address=45.91.116.8:4047 , piece_id=S5Z6E7DB7LLJLZ2TI6ZMHL6OI4ZQYNWHDZEPIN2H2IJ4T2BZF3JA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.068357320 (60h5m3.79092037s) [noise=true , address=45.91.116.13:4155 , piece_id=N6S2QTH5N3XXOJ3VL6AXMVP37MALI7MNAAJCMUZ5FLUTKE564M3A, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:55.813245504 (60h5m5.535808544s) [upload=57643]: Next returned nil segment Jul 26 23:40:55.813263219 (60h5m5.535826259s) [upload=57643]: Waiting for error group managing segments... Jul 26 23:40:56.090611589 (60h5m5.81317463s) [noise=true , address=45.91.116.8:4045 , piece_id=K6EU6IWY2C4JNFKE3GEHO2TXITUR4NDR36YNQ4DT574N3TDVX2YA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.095243321 (60h5m5.817806361s) [noise=true , address=45.91.116.10:4083 , piece_id=XIPEAMLQ7XASULSDU4IYGFZODUMXKWC4HIBDTSF57ECREBALVMYA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.097127957 (60h5m5.819690997s) [noise=true , address=45.91.116.13:4155 , piece_id=N6S2QTH5N3XXOJ3VL6AXMVP37MALI7MNAAJCMUZ5FLUTKE564M3A, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.176113910 (60h5m5.898676951s) [noise=true , address=45.91.116.8:4047 , piece_id=S5Z6E7DB7LLJLZ2TI6ZMHL6OI4ZQYNWHDZEPIN2H2IJ4T2BZF3JA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.176198943 (60h5m5.898761983s) [noise=true , address=45.91.116.12:4225 , piece_id=W3AJYCVJZ5X7O3M3ERKIUJRVBKQBZ5JPBQI2OQP2ZUDBGDMGKP3Q, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.202081505 (60h5m5.924644575s) [noise=true , address=45.91.116.4:4938 , piece_id=ZBHBICLFFLQZCUU5M6LPN7RBQR2ABGVC6XWID3ZNXF5KTJ22LBDQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.227513815 (60h5m5.950076855s) [noise=true , address=45.91.116.6:4973 , piece_id=HSQIBXKQNP3LTRF6SPEYXJVUHLT3Q3EB6XSOPA2AQ7N3SID73AFA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.257081847 (60h5m5.979644887s) [noise=true , address=45.91.116.11:4118 , piece_id=7QQIPZSUOTQ6SULT4UP2CT5JXZE2DGTOWKCEJ54KLYD573GPFDSQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:41:23.158146731 (60h5m32.880709771s) [noise=true , address=104.36.82.7:4912 , piece_id=C3NV4RSZGQZYX7WOBXL3QQJPFNLGVYFQVYBOAU2ZWE5EBZGXIF5A, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158190023 (60h5m32.880753063s) [noise=true , address=104.36.82.8:4969 , piece_id=YEXVSD6POLNBCC6QWSBVAT2CBCPPRAUOBTOT2MJXQQ3SNFTBTIJQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:36076->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158205493 (60h5m32.880768533s) [noise=true , address=104.36.82.13:4187 , piece_id=G3VQ6UIIWQXTPFZYFGQ25HWODTBAQZYWVVAJWOLOS5ALNJGDAHBA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 188.166.225.170:51644->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158245880 (60h5m32.88080892s) [noise=true , address=104.36.82.5:4839 , piece_id=4NVOW5SCB6357X4OC2LBN5UDKN73JD3RACG6KTFG7LUC3ECD5MEQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158274154 (60h5m32.880837194s) [noise=true , address=104.36.82.10:4078 , piece_id=5KZ3MYPM3QI6DEHKSFRVCMHBV3XLTBRDSPSTBHOFUTCJFHVTLWZQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158282551 (60h5m32.880845581s) [noise=true , address=104.36.82.9:4007 , piece_id=P4ZZAX4H7SNII47EWOF4JZCZ2CCTGXJFXHVBFXYFF5Q6KXHRR6WQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:60692->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158292049 (60h5m32.880855089s) [noise=true , address=104.36.82.16:4293 , piece_id=VXLOQOHR7ZOX6QV4K252TARFU7ERB7WSTLO6PQR3LVNK3PWOSNSQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158309132 (60h5m32.880872172s) [noise=true , address=104.36.82.4:4802 , piece_id=SRNVGYMPYK7BJBX2MZGZ3CRTZ6O4TCY6TTHDSUUIXZHCNLOBLMFQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:57370->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158325593 (60h5m32.880888623s) [seg_pos={0 0}, upload=57643]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:41:23.389168695 (60h5m33.111731735s) [upload=57643]: Done uploading segments... Jul 26 23:41:23.389195697 (60h5m33.111758727s) [upload=57643]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.233327272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.23265294s FAIL (2025-07-26 23:41:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232713516s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.831603045s FAIL (2025-07-26 23:41:13 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.831624766s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X....] duration: 38.739162725s FAIL (2025-07-26 23:41:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.739183824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236461143s FAIL (2025-07-26 23:41:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== next segment. Inline: false Jul 26 23:40:01.891301178 (60h4m11.613864228s) [upload=57631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:40:01.891344101 (60h4m11.613907151s) [upload=57631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:40:02.334018641 (60h4m12.056581701s) [seg_pos={0 0}, upload=57631]: Begin upload segment... Jul 26 23:40:02.334094336 (60h4m12.056657386s) [seg_pos={0 0}, upload=57631]: Done begin upload segment. Jul 26 23:40:02.334124964 (60h4m12.056688014s) [noise=true , address=79.127.201.215:4358 , piece_id=H3CFX2TNM5WOEP6OO63HNH3TXTHH6I36AUZMSN2FIS7OWBVL62AA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334598690 (60h4m12.05716174s) [noise=true , address=64.49.13.140:4514 , piece_id=HB5MLNZGGJBJFQUMI7C7XDAMJWFYYSUK55Q5SSRJ3YPRKSHJHIMA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334817048 (60h4m12.057380088s) [noise=true , address=79.127.219.51:4196 , piece_id=NKFK5YHDYJDGRT76JDACGPCEEMHNNMVDW4JCDM2WQ4Q2XXF77CQQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334970051 (60h4m12.057533101s) [noise=true , address=104.36.82.55:4228 , piece_id=JP5LCG653OM4IBJZYK2E7LXLFPD4MD4LGLEY74NOYMKR2ORASTCQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335327766 (60h4m12.057890816s) [noise=true , address=79.127.205.235:4691 , piece_id=HBH3UHRNUF2WUOHIMDN2HF6YDK7USXRM66KQZVX53CI2LMDNXJQA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335352513 (60h4m12.057915563s) [noise=true , address=104.36.82.53:4140 , piece_id=GD422AISDOMSTCWJ33TAOQYYP2YOPEVCEDITIF5DEIGBN3RGXLGQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335670842 (60h4m12.058233892s) [noise=true , address=104.36.82.56:4259 , piece_id=CRKOTNKL4DZEYFVZIFMMRTI6YQP5IUN32IU3UM7TI254PHDM54EA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335681172 (60h4m12.058244212s) [noise=true , address=64.49.13.138:4483 , piece_id=MSQ2TKY75KQQ6IWOUOBF23Z7UZY6IUSEIHL3SEBQJI3VCC6KYGPQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336019499 (60h4m12.058582549s) [noise=true , address=64.49.13.138:4482 , piece_id=JG6NCXMZZ65SPWD6QPIVHQXSJVHCHVX5O3ARDBZNBEUQ35ZZMVNA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336288354 (60h4m12.058851404s) [noise=true , address=77.90.31.12:4611 , piece_id=W2HGHPCQ3SCRZXAMDGKPMTKQ5GUW6MBYBEOKGKMDBJIBLRORIV4Q, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336492574 (60h4m12.059055624s) [noise=true , address=79.127.205.245:4249 , piece_id=GUUNLUR2FAES7EYCY3LT4XB3EA4JVV7ZDGS37T7XLT6NOZTUFGBA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336667600 (60h4m12.05923065s) [noise=true , address=104.36.82.47:4548 , piece_id=GS7HYU3RGLABF7RRQNKUSESTYR2GUDAYNTSAM6JRQFR2OOESDDVQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337004233 (60h4m12.059567273s) [noise=true , address=79.127.205.234:4618 , piece_id=C7Q5X2K4GLJRWSBLY4EK7ZKKUSPQX43EMGW7XXFVFYMEPZAAISZA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337404198 (60h4m12.059967238s) [noise=true , address=104.36.82.54:4170 , piece_id=EKAXD6L76FPRQMPNTDLYYUKETWVCGIIKYH5UKF3KJPFG5HURKEKQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337693081 (60h4m12.060256121s) [noise=true , address=79.127.219.38:4009 , piece_id=TI6JLARWL77E2TSAPNXOGCUWCHAW42746NQHJLA7EERSAEAXFKYQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338041528 (60h4m12.060604578s) [noise=true , address=104.36.82.20:4447 , piece_id=GRDH4FZBYAEOIYZ7SKG2QJUCBFMPIF4PHXF3OAEZ2CT2TDK7XOTQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338435281 (60h4m12.060998321s) [noise=true , address=45.91.116.14:4206 , piece_id=4W4IYEOTJ2ECZR67RFJUB4CPDEAFAKADKBSVS5B3SGYZZVTE6GPA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338792094 (60h4m12.061355144s) [noise=true , address=45.91.116.12:4230 , piece_id=AX3DJTKNF7TGIIOTZZGIPZDOR7CUJVI73IRRLGCOFAHZ4AB35Y3A, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:03.435253225 (60h4m13.157816275s) [upload=57631]: Next returned nil segment Jul 26 23:40:03.435271992 (60h4m13.157835032s) [upload=57631]: Waiting for error group managing segments... Jul 26 23:40:03.713687615 (60h4m13.436250665s) [noise=true , address=79.127.205.245:4249 , piece_id=GUUNLUR2FAES7EYCY3LT4XB3EA4JVV7ZDGS37T7XLT6NOZTUFGBA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.724736384 (60h4m13.447299424s) [noise=true , address=79.127.205.235:4691 , piece_id=HBH3UHRNUF2WUOHIMDN2HF6YDK7USXRM66KQZVX53CI2LMDNXJQA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.745523066 (60h4m13.468086106s) [noise=true , address=64.49.13.140:4514 , piece_id=HB5MLNZGGJBJFQUMI7C7XDAMJWFYYSUK55Q5SSRJ3YPRKSHJHIMA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.776899468 (60h4m13.499462518s) [noise=true , address=79.127.205.234:4618 , piece_id=C7Q5X2K4GLJRWSBLY4EK7ZKKUSPQX43EMGW7XXFVFYMEPZAAISZA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.196053668 (60h4m13.918616718s) [noise=true , address=79.127.219.51:4196 , piece_id=NKFK5YHDYJDGRT76JDACGPCEEMHNNMVDW4JCDM2WQ4Q2XXF77CQQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.240435483 (60h4m13.962998533s) [noise=true , address=79.127.219.38:4009 , piece_id=TI6JLARWL77E2TSAPNXOGCUWCHAW42746NQHJLA7EERSAEAXFKYQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.344412083 (60h4m14.066975123s) [noise=true , address=77.90.31.12:4611 , piece_id=W2HGHPCQ3SCRZXAMDGKPMTKQ5GUW6MBYBEOKGKMDBJIBLRORIV4Q, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:05.211183609 (60h4m14.933746649s) [noise=true , address=64.49.13.138:4482 , piece_id=JG6NCXMZZ65SPWD6QPIVHQXSJVHCHVX5O3ARDBZNBEUQ35ZZMVNA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:08.895379238 (60h4m18.617942268s) [noise=true , address=64.49.13.138:4483 , piece_id=MSQ2TKY75KQQ6IWOUOBF23Z7UZY6IUSEIHL3SEBQJI3VCC6KYGPQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:10.854276141 (60h4m20.576839181s) [noise=true , address=45.91.116.12:4230 , piece_id=AX3DJTKNF7TGIIOTZZGIPZDOR7CUJVI73IRRLGCOFAHZ4AB35Y3A, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:11.901007675 (60h4m21.623570715s) [noise=true , address=45.91.116.14:4206 , piece_id=4W4IYEOTJ2ECZR67RFJUB4CPDEAFAKADKBSVS5B3SGYZZVTE6GPA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:42.400223133 (60h4m52.122786173s) [noise=true , address=79.127.201.215:4358 , piece_id=H3CFX2TNM5WOEP6OO63HNH3TXTHH6I36AUZMSN2FIS7OWBVL62AA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:41:01.238787306 (60h5m10.961350356s) [noise=true , address=104.36.82.53:4140 , piece_id=GD422AISDOMSTCWJ33TAOQYYP2YOPEVCEDITIF5DEIGBN3RGXLGQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: rpc: dial tcp 104.36.82.53:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.238869393 (60h5m10.961432433s) [noise=true , address=104.36.82.56:4259 , piece_id=CRKOTNKL4DZEYFVZIFMMRTI6YQP5IUN32IU3UM7TI254PHDM54EA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:41:01.238917414 (60h5m10.961480454s) [noise=true , address=104.36.82.47:4548 , piece_id=GS7HYU3RGLABF7RRQNKUSESTYR2GUDAYNTSAM6JRQFR2OOESDDVQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.238967710 (60h5m10.96153075s) [noise=true , address=104.36.82.20:4447 , piece_id=GRDH4FZBYAEOIYZ7SKG2QJUCBFMPIF4PHXF3OAEZ2CT2TDK7XOTQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1Li8ZwrWGPx5Y478pYLwnWJ7N2x3dbXgwGaUGWVtVhvv61EnTs, address:104.36.82.20:4447): protocol: write tcp 188.166.225.170:51350->104.36.82.20:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239026923 (60h5m10.961589964s) [noise=true , address=104.36.82.54:4170 , piece_id=EKAXD6L76FPRQMPNTDLYYUKETWVCGIIKYH5UKF3KJPFG5HURKEKQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: rpc: dial tcp 104.36.82.54:4170: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239067111 (60h5m10.961630151s) [noise=true , address=104.36.82.55:4228 , piece_id=JP5LCG653OM4IBJZYK2E7LXLFPD4MD4LGLEY74NOYMKR2ORASTCQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 188.166.225.170:40190->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239095826 (60h5m10.961658866s) [seg_pos={0 0}, upload=57631]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:41:01.473296372 (60h5m11.195859422s) [upload=57631]: Done uploading segments... Jul 26 23:41:01.473375273 (60h5m11.195938323s) [upload=57631]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.236478415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X] duration: 30.000445402s FAIL (2025-07-26 23:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000462215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235260476s FAIL (2025-07-26 23:40:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235283051s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.......X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232650781s FAIL (2025-07-26 23:40:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== [upload=57629]: Done starting upload Jul 26 23:39:54.613218404 (60h4m4.335781494s) [upload=57629]: Uploading segments... Jul 26 23:39:54.613596457 (60h4m4.336159538s) [upload=57629]: Got next segment. Inline: false Jul 26 23:39:54.613683073 (60h4m4.336246163s) [upload=57629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:54.613691819 (60h4m4.33625489s) [upload=57629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:54.847068078 (60h4m4.569631118s) [seg_pos={0 0}, upload=57629]: Begin upload segment... Jul 26 23:39:54.847140146 (60h4m4.569703186s) [seg_pos={0 0}, upload=57629]: Done begin upload segment. Jul 26 23:39:54.847171676 (60h4m4.569734716s) [noise=true , address=104.36.82.12:4150 , piece_id=OID3AOLTTEGHC7FI4ABETMTI5TIM7IALBJHE3IX2DFBA756LAMQQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.847501036 (60h4m4.570064076s) [noise=true , address=104.36.82.16:4295 , piece_id=T2Z3M7JYTMCT3X2AHFDMHCWBV23T33BKOUMUJY4S4THMBGYQDROQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.847747057 (60h4m4.570310087s) [noise=true , address=45.91.116.7:4011 , piece_id=X6LZCMNYIWDQF3FCZXE56MWYZEMBJ4BFZGXDPDG4HDPCQORZYYPQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848023155 (60h4m4.570586195s) [noise=true , address=104.36.82.7:4912 , piece_id=HWKVVPMIBSYGOVFRDXZS2NCSIWQZH4LJUJ3V3SDME2QDOR2TQM6A, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848387672 (60h4m4.570950713s) [noise=true , address=77.90.31.18:4806 , piece_id=EZ2YVAHV6AA7OZWKHQG6J5ULGYXMQXLVWVHRASRLPQ2LDJG5IA7Q, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848730528 (60h4m4.571293568s) [noise=true , address=45.91.116.13:4153 , piece_id=YKW7X3J77MD255KUBGSBPKIT66CHMTTNIHCYIUVYQMLPR6Z2P5PQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849008911 (60h4m4.571571951s) [noise=true , address=104.36.82.5:4840 , piece_id=X35OIBIFJVBKWZ2TS4MBCA6J4GKYICZTMH5ZOEODF6IDONHYIAAA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849216578 (60h4m4.571779618s) [noise=true , address=45.91.116.6:4974 , piece_id=ZPCKVLCI4TJVXJHCL24Z5NKTD775HFBUND4UEMHHHZN675FY4HHQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849531390 (60h4m4.57209443s) [noise=true , address=45.91.116.5:4297 , piece_id=IOAI7IFMLDABT2H7RSCR3TQH7QA7PVTM4FSSJ7KLEI5ELHR337PA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849739569 (60h4m4.572302599s) [noise=true , address=104.36.82.2:4732 , piece_id=NHZWL4O44AKJ2GIZ6KDO5KM66JIM42TZTXZR3VWJKIHFLWQJ4GCQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849871912 (60h4m4.572434952s) [noise=true , address=104.36.82.9:4007 , piece_id=N2M5H3LFJWZKMGKRCHLANERL2MBT7LERMRECWVKBPEQB2GCVGPIA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850145285 (60h4m4.572708325s) [noise=true , address=45.91.116.10:4081 , piece_id=ADABOJYSIWFVQNDFEKBJI3DKJAETE2NWAB36IM7NKKLL3JYSKC5A, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850517116 (60h4m4.573080156s) [noise=true , address=104.36.82.14:4223 , piece_id=DEWIE4BGTOCEFQ3MRMQTPL34K7MPH4QNO3OYKINH4DXSNJJR7CIA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850860484 (60h4m4.573423534s) [noise=true , address=104.36.82.3:4767 , piece_id=K3J4WV5GD2MYO2ZZ6MKDTOGM3PBKJENJ2EGHYDFI55BFAZW5V7IQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.851112666 (60h4m4.573675706s) [noise=true , address=77.90.31.13:4627 , piece_id=VR7ZYG7CAY26LNQB4KXSLZ67AQ3YCNYZSDSPQUAJH4VAUUWMSB4Q, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.851328700 (60h4m4.57389174s) [noise=true , address=45.91.116.4:4940 , piece_id=3BZU4G5NW3OXGY7ASUPAS4CT5MBY5D7LSZON6C5HRRCOFJXURNSA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:56.571267441 (60h4m6.293830492s) [upload=57629]: Next returned nil segment Jul 26 23:39:56.571287309 (60h4m6.293850349s) [upload=57629]: Waiting for error group managing segments... Jul 26 23:39:56.861708441 (60h4m6.584271471s) [noise=true , address=45.91.116.6:4974 , piece_id=ZPCKVLCI4TJVXJHCL24Z5NKTD775HFBUND4UEMHHHZN675FY4HHQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.869633655 (60h4m6.592196695s) [noise=true , address=45.91.116.4:4940 , piece_id=3BZU4G5NW3OXGY7ASUPAS4CT5MBY5D7LSZON6C5HRRCOFJXURNSA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.962593204 (60h4m6.685156244s) [noise=true , address=45.91.116.13:4153 , piece_id=YKW7X3J77MD255KUBGSBPKIT66CHMTTNIHCYIUVYQMLPR6Z2P5PQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.963153938 (60h4m6.685716978s) [noise=true , address=77.90.31.18:4806 , piece_id=EZ2YVAHV6AA7OZWKHQG6J5ULGYXMQXLVWVHRASRLPQ2LDJG5IA7Q, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.965557095 (60h4m6.688120125s) [noise=true , address=45.91.116.7:4011 , piece_id=X6LZCMNYIWDQF3FCZXE56MWYZEMBJ4BFZGXDPDG4HDPCQORZYYPQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:57.045039820 (60h4m6.76760286s) [noise=true , address=45.91.116.10:4081 , piece_id=ADABOJYSIWFVQNDFEKBJI3DKJAETE2NWAB36IM7NKKLL3JYSKC5A, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:57.083542838 (60h4m6.806105878s) [noise=true , address=45.91.116.5:4297 , piece_id=IOAI7IFMLDABT2H7RSCR3TQH7QA7PVTM4FSSJ7KLEI5ELHR337PA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:58.302482007 (60h4m8.025045047s) [noise=true , address=77.90.31.13:4627 , piece_id=VR7ZYG7CAY26LNQB4KXSLZ67AQ3YCNYZSDSPQUAJH4VAUUWMSB4Q, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:40:23.972358768 (60h4m33.694921808s) [noise=true , address=104.36.82.3:4767 , piece_id=K3J4WV5GD2MYO2ZZ6MKDTOGM3PBKJENJ2EGHYDFI55BFAZW5V7IQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:42548->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972425154 (60h4m33.694988205s) [noise=true , address=104.36.82.7:4912 , piece_id=HWKVVPMIBSYGOVFRDXZS2NCSIWQZH4LJUJ3V3SDME2QDOR2TQM6A, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:39050->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972494999 (60h4m33.695058059s) [noise=true , address=104.36.82.5:4840 , piece_id=X35OIBIFJVBKWZ2TS4MBCA6J4GKYICZTMH5ZOEODF6IDONHYIAAA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:55986->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972672428 (60h4m33.695235478s) [noise=true , address=104.36.82.2:4732 , piece_id=NHZWL4O44AKJ2GIZ6KDO5KM66JIM42TZTXZR3VWJKIHFLWQJ4GCQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972730348 (60h4m33.695293388s) [noise=true , address=104.36.82.14:4223 , piece_id=DEWIE4BGTOCEFQ3MRMQTPL34K7MPH4QNO3OYKINH4DXSNJJR7CIA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:53096->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972784672 (60h4m33.695347732s) [noise=true , address=104.36.82.12:4150 , piece_id=OID3AOLTTEGHC7FI4ABETMTI5TIM7IALBJHE3IX2DFBA756LAMQQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:45708->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972819510 (60h4m33.69538257s) [noise=true , address=104.36.82.16:4295 , piece_id=T2Z3M7JYTMCT3X2AHFDMHCWBV23T33BKOUMUJY4S4THMBGYQDROQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:56532->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972877450 (60h4m33.69544049s) [noise=true , address=104.36.82.9:4007 , piece_id=N2M5H3LFJWZKMGKRCHLANERL2MBT7LERMRECWVKBPEQB2GCVGPIA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:58560->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972922356 (60h4m33.695485406s) [seg_pos={0 0}, upload=57629]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:40:24.202943213 (60h4m33.925506313s) [upload=57629]: Done uploading segments... Jul 26 23:40:24.202967600 (60h4m33.92553064s) [upload=57629]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.232663224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.829142391s FAIL (2025-07-26 23:40:12 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.829165185s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X...] duration: 42.683228493s FAIL (2025-07-26 23:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.683241828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.22957946s FAIL (2025-07-26 23:39:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== upload=57617]: Begin upload segment... Jul 26 23:39:00.563533224 (60h3m10.286096264s) [seg_pos={0 0}, upload=57617]: Done begin upload segment. Jul 26 23:39:00.563555096 (60h3m10.286118136s) [noise=true , address=79.127.201.215:4358 , piece_id=OPORJ3WXI6TSU2SMTWLJUT46LH5PGU2VIUWDBHMN7D3PIVBUIHDQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.563617165 (60h3m10.286180205s) [noise=true , address=45.91.116.14:4194 , piece_id=HNEDBA7KJKSUGQQGJZ7MPAD6X5U5REJFHPOI5ULX3SBXPPQFXHSQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.563949611 (60h3m10.286512651s) [noise=true , address=104.36.82.14:4241 , piece_id=H6GYPIUJ7YIBWR5L627PHSZGW5FFKQOO2IHFIXTLTRTAY63YEDHA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564133212 (60h3m10.286696252s) [noise=true , address=64.49.13.138:4483 , piece_id=OH3CTMXK3DVWLN7WNAQ5FLP2K3JH6OIPATAK2Y6JN44XSMAZ3AAQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564296865 (60h3m10.286859906s) [noise=true , address=79.127.205.234:4606 , piece_id=AAHNXSTSO6H4VAC4TBVZGZXGNFZIY6IBR6WT6HYOJAXUMMQYNUGA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564548687 (60h3m10.287111727s) [noise=true , address=79.127.205.246:4262 , piece_id=RL6WH366VAZ24OVWMRATCUMSCNLJCR6HNASEEKOWCCBP6VTJ7MXQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564577853 (60h3m10.287140893s) [noise=true , address=45.91.116.9:4277 , piece_id=4LL6JDY53XTTHQJLQP325DP7KYF6ISYVDJTEWU6OFCP7MAPYM5VQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564806290 (60h3m10.28736934s) [noise=true , address=104.36.82.54:4173 , piece_id=D5AHPXYPRAO6IDETDNBPSARU67DS3UBJLV7X532MCESUSEEZSGUA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564888347 (60h3m10.287451387s) [noise=true , address=79.127.201.215:4355 , piece_id=3PWIAFHOLA6OESPI6AKJW35FLVXI6KU2YIOW55E5JEZIIJ2GNK5Q, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565031360 (60h3m10.2875944s) [noise=true , address=104.36.82.51:4060 , piece_id=K2NH4DHMZAVUULNGZ36HYG43QQEPRJ5SR7QRERZ37VPJVP2HIOFA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565139717 (60h3m10.287702758s) [noise=true , address=104.36.82.46:4508 , piece_id=KBXR3IQDB2TDYEAXBJLXHUHCIPUDLPARSXCND75HMMEGF4LYNBAQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565264657 (60h3m10.287827697s) [noise=true , address=104.36.82.53:4128 , piece_id=4B6DJQ4ZD56EFO2HYEJOQQRAHJZMN2NVXFGYOBJ5RLQITVAF4WDA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565440523 (60h3m10.288003553s) [noise=true , address=77.90.31.10:4540 , piece_id=3YSXDHGJP44FOOWMG3LJLPTOYJLDCISXREIRLG53GEU32GWSZ55A, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565480280 (60h3m10.28804332s) [noise=true , address=79.127.205.245:4244 , piece_id=FKXN2W6NZIQ24Q5WG3XBCSDRAP77Y73ZJDNGHE3IANEJB4SYYXGA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565681094 (60h3m10.288244134s) [noise=true , address=45.91.116.9:4273 , piece_id=UX7QDOCZR3THS7AZOMQAYQSYFKJLQGYRIGIMWLDSWTZAKSDQALTQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565838465 (60h3m10.288401505s) [noise=true , address=77.90.31.12:4599 , piece_id=QJZOQ4YNLEP4Y5ILOFSNPDKG6NKOPEX3MRF4UU4I7GBGYQ2RKLCQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.566095657 (60h3m10.288658697s) [noise=true , address=77.90.31.11:4559 , piece_id=WN3CHPURJOHLSHQGCSUXQGRNDKGAI7EKS6GYV3L6S2U6WAEM2PIQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.569345735 (60h3m10.291908785s) [noise=true , address=156.146.43.225:4449 , piece_id=LUBMQ6EFDU5775O7HUT3YJZT6FRS5XI7S47ITF6DVEO6PNMSCQWQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:01.590872054 (60h3m11.313435094s) [noise=true , address=156.146.43.225:4449 , piece_id=LUBMQ6EFDU5775O7HUT3YJZT6FRS5XI7S47ITF6DVEO6PNMSCQWQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:39:01.664029455 (60h3m11.386592495s) [upload=57617]: Next returned nil segment Jul 26 23:39:01.664046087 (60h3m11.386609127s) [upload=57617]: Waiting for error group managing segments... Jul 26 23:39:01.942806661 (60h3m11.665369701s) [noise=true , address=79.127.205.234:4606 , piece_id=AAHNXSTSO6H4VAC4TBVZGZXGNFZIY6IBR6WT6HYOJAXUMMQYNUGA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:01.950304697 (60h3m11.672867737s) [noise=true , address=79.127.205.246:4262 , piece_id=RL6WH366VAZ24OVWMRATCUMSCNLJCR6HNASEEKOWCCBP6VTJ7MXQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:01.966901802 (60h3m11.689464842s) [noise=true , address=79.127.205.245:4244 , piece_id=FKXN2W6NZIQ24Q5WG3XBCSDRAP77Y73ZJDNGHE3IANEJB4SYYXGA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:02.531205896 (60h3m12.253768936s) [noise=true , address=45.91.116.9:4277 , piece_id=4LL6JDY53XTTHQJLQP325DP7KYF6ISYVDJTEWU6OFCP7MAPYM5VQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:02.566336792 (60h3m12.288899832s) [noise=true , address=77.90.31.12:4599 , piece_id=QJZOQ4YNLEP4Y5ILOFSNPDKG6NKOPEX3MRF4UU4I7GBGYQ2RKLCQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:03.693300014 (60h3m13.415863055s) [noise=true , address=45.91.116.9:4273 , piece_id=UX7QDOCZR3THS7AZOMQAYQSYFKJLQGYRIGIMWLDSWTZAKSDQALTQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:07.489158787 (60h3m17.211721857s) [noise=true , address=45.91.116.14:4194 , piece_id=HNEDBA7KJKSUGQQGJZ7MPAD6X5U5REJFHPOI5ULX3SBXPPQFXHSQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:11.678653795 (60h3m21.401216835s) [noise=true , address=64.49.13.138:4483 , piece_id=OH3CTMXK3DVWLN7WNAQ5FLP2K3JH6OIPATAK2Y6JN44XSMAZ3AAQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:13.481650041 (60h3m23.204213081s) [noise=true , address=79.127.201.215:4355 , piece_id=3PWIAFHOLA6OESPI6AKJW35FLVXI6KU2YIOW55E5JEZIIJ2GNK5Q, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:32.586856572 (60h3m42.309419602s) [noise=true , address=79.127.201.215:4358 , piece_id=OPORJ3WXI6TSU2SMTWLJUT46LH5PGU2VIUWDBHMN7D3PIVBUIHDQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:59.472484714 (60h4m9.195047754s) [noise=true , address=77.90.31.10:4540 , piece_id=3YSXDHGJP44FOOWMG3LJLPTOYJLDCISXREIRLG53GEU32GWSZ55A, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:39:59.472597730 (60h4m9.19516076s) [noise=true , address=104.36.82.54:4173 , piece_id=D5AHPXYPRAO6IDETDNBPSARU67DS3UBJLV7X532MCESUSEEZSGUA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472687112 (60h4m9.195250152s) [noise=true , address=104.36.82.46:4508 , piece_id=KBXR3IQDB2TDYEAXBJLXHUHCIPUDLPARSXCND75HMMEGF4LYNBAQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:39:59.472742578 (60h4m9.195305608s) [noise=true , address=104.36.82.53:4128 , piece_id=4B6DJQ4ZD56EFO2HYEJOQQRAHJZMN2NVXFGYOBJ5RLQITVAF4WDA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: rpc: dial tcp 104.36.82.53:4128: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472831308 (60h4m9.195394348s) [noise=true , address=104.36.82.14:4241 , piece_id=H6GYPIUJ7YIBWR5L627PHSZGW5FFKQOO2IHFIXTLTRTAY63YEDHA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472872005 (60h4m9.195435045s) [noise=true , address=104.36.82.51:4060 , piece_id=K2NH4DHMZAVUULNGZ36HYG43QQEPRJ5SR7QRERZ37VPJVP2HIOFA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:19VJKnTAges9YKYdUVoz3tb3sFU1bawxpaJ2rsBd2wmLo1zK4X, address:104.36.82.51:4060): protocol: write tcp 188.166.225.170:50200->104.36.82.51:4060: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472895179 (60h4m9.19545822s) [noise=true , address=77.90.31.11:4559 , piece_id=WN3CHPURJOHLSHQGCSUXQGRNDKGAI7EKS6GYV3L6S2U6WAEM2PIQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: read tcp 188.166.225.170:53148->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:39:59.472914677 (60h4m9.195477717s) [seg_pos={0 0}, upload=57617]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:39:59.700511699 (60h4m9.423074739s) [upload=57617]: Done uploading segments... Jul 26 23:39:59.700537529 (60h4m9.423100569s) [upload=57617]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.229592836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237310537s FAIL (2025-07-26 23:39:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== d enough pieces (needed at least 13 but got 4) Jul 26 23:39:24.147360527 (60h3m33.869923567s) [upload=57624]: Starting upload Jul 26 23:39:24.147422626 (60h3m33.869985666s) [upload=57624]: Done starting upload Jul 26 23:39:24.147510373 (60h3m33.870073414s) [upload=57624]: Uploading segments... Jul 26 23:39:24.147595657 (60h3m33.870158697s) [upload=57624]: Got next segment. Inline: false Jul 26 23:39:24.147621466 (60h3m33.870184506s) [upload=57624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:24.147625765 (60h3m33.870188805s) [upload=57624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:24.476822648 (60h3m34.199385688s) [seg_pos={0 0}, upload=57624]: Begin upload segment... Jul 26 23:39:24.476871762 (60h3m34.199434802s) [seg_pos={0 0}, upload=57624]: Done begin upload segment. Jul 26 23:39:24.476894746 (60h3m34.199457776s) [noise=true , address=104.36.82.4:4804 , piece_id=GFBUBQ2WZVH7DKBMJ26DG4BVFGFZXKBKNLTQPXFZE3E32EW4IXIA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.476910366 (60h3m34.199473406s) [noise=true , address=104.36.82.13:4187 , piece_id=S7HJZC2V3TOOUNVBRHUCT47URKQ6EA3XDBDFNWNPWKS6MNJI5MGA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477231360 (60h3m34.19979439s) [noise=true , address=104.36.82.3:4767 , piece_id=CJDCRY3JYY7XV6VBSRVJCSAWUIM7KAD6OAVYFHVDTZWWSPOTLR5A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477234686 (60h3m34.199797726s) [noise=true , address=45.91.116.4:4938 , piece_id=PR5XK5SF7WIC3QGCOFFHIDG5CYNF3TWBHUPYQ46VZK7HY6XGLZTA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477485145 (60h3m34.200048185s) [noise=true , address=45.91.116.8:4047 , piece_id=VRIUESCP4JHUXIFA73F244RPKOPRZC56JENBLKZBFNONJHN7EDFQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477491468 (60h3m34.200054508s) [noise=true , address=45.91.116.10:4083 , piece_id=P3ID2XHSJHUSIGL3HSKCLXPQQVGW6K2UCEDAV5XKNAEY5G67Z52A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477738511 (60h3m34.200301541s) [noise=true , address=104.36.82.5:4839 , piece_id=W7MD6NGGBGKJWPLQTZXF2EX233K54WK4MI5DMGALW6QTED4XCHJA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477745805 (60h3m34.200308835s) [noise=true , address=77.90.31.14:4661 , piece_id=MSRPMZQZXRK2FC5U6ISJPGMMGUOASHJYTLRJYUGYQ4RB4F5HXEAQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477971576 (60h3m34.200534616s) [noise=true , address=45.91.116.7:4011 , piece_id=RDUDNWPDME2ASGGGNJSDAX3I3QRRXIG27PAOVB3M5I4MM2DVJRLQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478067841 (60h3m34.200630881s) [noise=true , address=104.36.82.12:4150 , piece_id=F4VXQSQCPSOND7BNAI3N6Y3CT7SBACEHUHS5WVM3IQ4L24RTBL4A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478227356 (60h3m34.200790396s) [noise=true , address=45.91.116.6:4973 , piece_id=VENHMAXJFIREXN5EYT5N6VE3VL5HCRNRPZSX3CNIHU5PHVIKRTMA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478305595 (60h3m34.200868625s) [noise=true , address=104.36.82.15:4259 , piece_id=5OSOZR6RWHAEWIU2ECCQMXZV2IBJBFQ5PRZAEEH32R5C7X55U3PA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478525386 (60h3m34.201088426s) [noise=true , address=77.90.31.19:4841 , piece_id=RQELPXBAA7OOG7VDJZYOGWB5VCLPNL6Y3R5CHM6VJ3DITTN6Q6ZQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478747010 (60h3m34.20131004s) [noise=true , address=45.91.116.5:4298 , piece_id=SHWJLIZ4NYKAYFDYXB2TJCFUSAEPW3DSJQSZNBBRWQA7HYXGYLCQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.479153227 (60h3m34.201716267s) [noise=true , address=104.36.82.14:4223 , piece_id=RH4OJUFBJWBMHA7HQ6WBJJ4MX7G6AI7RUNVL4MHPUV5RHY55YSRA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.479525179 (60h3m34.202088219s) [noise=true , address=104.36.82.6:4873 , piece_id=FBWXVEFQ65ZUZV636WDYOHIXKTHQBK7LBGT6735GQN56XZ4UM2WQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:26.192582066 (60h3m35.915145106s) [upload=57624]: Next returned nil segment Jul 26 23:39:26.192598226 (60h3m35.915161266s) [upload=57624]: Waiting for error group managing segments... Jul 26 23:39:26.488939757 (60h3m36.211502797s) [noise=true , address=45.91.116.4:4938 , piece_id=PR5XK5SF7WIC3QGCOFFHIDG5CYNF3TWBHUPYQ46VZK7HY6XGLZTA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.502341068 (60h3m36.224904098s) [noise=true , address=45.91.116.5:4298 , piece_id=SHWJLIZ4NYKAYFDYXB2TJCFUSAEPW3DSJQSZNBBRWQA7HYXGYLCQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.510343499 (60h3m36.232906539s) [noise=true , address=45.91.116.7:4011 , piece_id=RDUDNWPDME2ASGGGNJSDAX3I3QRRXIG27PAOVB3M5I4MM2DVJRLQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.589409295 (60h3m36.311972335s) [noise=true , address=77.90.31.14:4661 , piece_id=MSRPMZQZXRK2FC5U6ISJPGMMGUOASHJYTLRJYUGYQ4RB4F5HXEAQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.589853746 (60h3m36.312416796s) [noise=true , address=45.91.116.6:4973 , piece_id=VENHMAXJFIREXN5EYT5N6VE3VL5HCRNRPZSX3CNIHU5PHVIKRTMA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.591676964 (60h3m36.314240004s) [noise=true , address=45.91.116.8:4047 , piece_id=VRIUESCP4JHUXIFA73F244RPKOPRZC56JENBLKZBFNONJHN7EDFQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.655348387 (60h3m36.377911427s) [noise=true , address=45.91.116.10:4083 , piece_id=P3ID2XHSJHUSIGL3HSKCLXPQQVGW6K2UCEDAV5XKNAEY5G67Z52A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:28.117305011 (60h3m37.839868051s) [noise=true , address=77.90.31.19:4841 , piece_id=RQELPXBAA7OOG7VDJZYOGWB5VCLPNL6Y3R5CHM6VJ3DITTN6Q6ZQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:53.503295136 (60h4m3.225858166s) [noise=true , address=104.36.82.15:4259 , piece_id=5OSOZR6RWHAEWIU2ECCQMXZV2IBJBFQ5PRZAEEH32R5C7X55U3PA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503348258 (60h4m3.225911288s) [noise=true , address=104.36.82.5:4839 , piece_id=W7MD6NGGBGKJWPLQTZXF2EX233K54WK4MI5DMGALW6QTED4XCHJA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:58420->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503452648 (60h4m3.226015688s) [noise=true , address=104.36.82.3:4767 , piece_id=CJDCRY3JYY7XV6VBSRVJCSAWUIM7KAD6OAVYFHVDTZWWSPOTLR5A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503485220 (60h4m3.22604826s) [noise=true , address=104.36.82.4:4804 , piece_id=GFBUBQ2WZVH7DKBMJ26DG4BVFGFZXKBKNLTQPXFZE3E32EW4IXIA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:34774->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503501671 (60h4m3.226064711s) [noise=true , address=104.36.82.12:4150 , piece_id=F4VXQSQCPSOND7BNAI3N6Y3CT7SBACEHUHS5WVM3IQ4L24RTBL4A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503527280 (60h4m3.22609032s) [noise=true , address=104.36.82.13:4187 , piece_id=S7HJZC2V3TOOUNVBRHUCT47URKQ6EA3XDBDFNWNPWKS6MNJI5MGA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503534935 (60h4m3.226097975s) [noise=true , address=104.36.82.14:4223 , piece_id=RH4OJUFBJWBMHA7HQ6WBJJ4MX7G6AI7RUNVL4MHPUV5RHY55YSRA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503566025 (60h4m3.226129065s) [noise=true , address=104.36.82.6:4873 , piece_id=FBWXVEFQ65ZUZV636WDYOHIXKTHQBK7LBGT6735GQN56XZ4UM2WQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503593728 (60h4m3.226156768s) [seg_pos={0 0}, upload=57624]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:39:53.738260936 (60h4m3.460823976s) [upload=57624]: Done uploading segments... Jul 26 23:39:53.738292727 (60h4m3.460855847s) [upload=57624]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.237340895s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.] duration: 27.7280688s FAIL (2025-07-26 23:39:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.728088998s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235054871s FAIL (2025-07-26 23:39:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23507538s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X..] duration: 46.905355123s FAIL (2025-07-26 23:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.905368018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.290008255s FAIL (2025-07-26 23:39:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.290019787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [X.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232243318s FAIL (2025-07-26 23:38:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== .82.6:4899 , piece_id=GHNDWYDAJ5ULHBZS5W566CUGVAQNAJYXSXNJ4Y7OKVUXQMJ3O2MA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116496262 (60h2m9.839059312s) [noise=true , address=77.90.31.18:4815 , piece_id=PPQFNBEH5J6R6YS2YHY63T4O7LJH7QPUFHCX6BGV5ZLUSZE6GRMQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116743264 (60h2m9.839306304s) [noise=true , address=77.90.31.21:4905 , piece_id=HHUOZXJUW6HHRB6PDZ6Z62K27DY27LB644UO4W2K5PI2HSJMH3NQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116901738 (60h2m9.839464778s) [noise=true , address=79.127.205.234:4613 , piece_id=EPLKZP6ZFV6SMXWQVQ5EAS37XBZZWI35ZXXHC77OUJPVZJ5T4MSA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116993653 (60h2m9.839556683s) [noise=true , address=79.127.219.38:4015 , piece_id=YIFJC6QCHKBOA7WJHJVEHNABNB55NSDTFAFWAMEZR7LGDRXPGFRQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117239413 (60h2m9.839802453s) [noise=true , address=79.127.205.235:4667 , piece_id=JZR6IERCRWL65D57LGXOLCK7FABAZK2CC3426PLHZVPO55EAZZTA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117485364 (60h2m9.840048404s) [noise=true , address=104.36.82.55:4224 , piece_id=ATSAO7QMB43NHCL4IGW2APVOACK7FHU4LMQ4OIELBRNBWXEG3VHQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117562742 (60h2m9.840125782s) [noise=true , address=104.36.82.15:4273 , piece_id=A4M2WE67BHCHFYK7VYIICVVCHOMQBJAUMFQVP3U3AFOBC6EVGSSQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117702249 (60h2m9.840265289s) [noise=true , address=104.36.82.56:4261 , piece_id=2A34GGYXGU3WH3CDZARWEBLUQUGDABNSZXYGJG4BNMUONCYFYHPQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117934844 (60h2m9.840497884s) [noise=true , address=79.127.201.215:4345 , piece_id=XCE3GOM2XSR5TE2F65IT7K7KVKC354IZ6WMSLCI6SLHWWT7BDDYQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118055555 (60h2m9.840618595s) [noise=true , address=79.127.220.97:4361 , piece_id=KZHYYS7FHM4ZAUBLXKL3D7EYNXJ3IUI7NTIRAP6JVI3ASTLMF72Q, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118364246 (60h2m9.840927286s) [noise=true , address=77.90.31.13:4638 , piece_id=OLKFQDFUBERTMW6PIRJJA4KO5R43IZ3RGLDQYCDW4TLLBOJPAWTQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118644291 (60h2m9.841207331s) [noise=true , address=64.49.13.138:4482 , piece_id=YJDJYD5GFH7Q27Z6GWX3SBSW2VMGXI5M4F3BWYU7EW3PCV322X7A, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:01.252588591 (60h2m10.975151631s) [upload=57602]: Next returned nil segment Jul 26 23:38:01.252606215 (60h2m10.975169255s) [upload=57602]: Waiting for error group managing segments... Jul 26 23:38:01.539602033 (60h2m11.262165063s) [noise=true , address=79.127.205.234:4613 , piece_id=EPLKZP6ZFV6SMXWQVQ5EAS37XBZZWI35ZXXHC77OUJPVZJ5T4MSA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:01.564041003 (60h2m11.286604043s) [noise=true , address=79.127.205.235:4667 , piece_id=JZR6IERCRWL65D57LGXOLCK7FABAZK2CC3426PLHZVPO55EAZZTA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.006438754 (60h2m11.729001794s) [noise=true , address=79.127.219.38:4015 , piece_id=YIFJC6QCHKBOA7WJHJVEHNABNB55NSDTFAFWAMEZR7LGDRXPGFRQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.118334845 (60h2m11.840897915s) [noise=true , address=77.90.31.18:4815 , piece_id=PPQFNBEH5J6R6YS2YHY63T4O7LJH7QPUFHCX6BGV5ZLUSZE6GRMQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.806564256 (60h2m12.529127296s) [noise=true , address=64.49.13.138:4482 , piece_id=YJDJYD5GFH7Q27Z6GWX3SBSW2VMGXI5M4F3BWYU7EW3PCV322X7A, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:38:05.670956897 (60h2m15.393519937s) [noise=true , address=79.127.220.97:4361 , piece_id=KZHYYS7FHM4ZAUBLXKL3D7EYNXJ3IUI7NTIRAP6JVI3ASTLMF72Q, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:09.006182463 (60h2m18.728745503s) [noise=true , address=79.127.201.215:4345 , piece_id=XCE3GOM2XSR5TE2F65IT7K7KVKC354IZ6WMSLCI6SLHWWT7BDDYQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:59.016355526 (60h3m8.738918606s) [noise=true , address=104.36.82.56:4261 , piece_id=2A34GGYXGU3WH3CDZARWEBLUQUGDABNSZXYGJG4BNMUONCYFYHPQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: write tcp 188.166.225.170:51566->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016434367 (60h3m8.738997407s) [noise=true , address=104.36.82.12:4183 , piece_id=WVC63ZGZVLBCVSQJHQKSP3HK3QW5MNYRJ3K54UWHZIRYVK3WS6FQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: write tcp 188.166.225.170:57746->104.36.82.12:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016476327 (60h3m8.739039367s) [noise=true , address=104.36.82.47:4527 , piece_id=ULI54HPWTC4WR3INH4OE5TSI5M3SKK5ZAF6MIOEWL6QJQLGQSQAQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 188.166.225.170:38294->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016495514 (60h3m8.739058544s) [noise=true , address=77.90.31.17:4801 , piece_id=7MNLNQK3ZD6SDGZKJFN4FAQ6V4FNVCKWAVJ76JCO47Z7VRL7QRPA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: write tcp 188.166.225.170:43164->77.90.31.17:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016510403 (60h3m8.739073443s) [noise=true , address=104.36.82.22:4521 , piece_id=QMNMNIELBPAA6NVS4N2J6V7QWO7HVO4YKOJS57NG5EE2T2RROQMA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1zWiwpaV58h8YKsAgw24fUWHBBYYyqF4UX6GJV316Uxo9Y7gt1, address:104.36.82.22:4521): protocol: write tcp 188.166.225.170:42360->104.36.82.22:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016542724 (60h3m8.739105764s) [noise=true , address=104.36.82.6:4899 , piece_id=GHNDWYDAJ5ULHBZS5W566CUGVAQNAJYXSXNJ4Y7OKVUXQMJ3O2MA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 188.166.225.170:38902->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016548655 (60h3m8.739111695s) [noise=true , address=77.90.31.13:4638 , piece_id=OLKFQDFUBERTMW6PIRJJA4KO5R43IZ3RGLDQYCDW4TLLBOJPAWTQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:38:59.016553785 (60h3m8.739116825s) [noise=true , address=104.36.82.55:4224 , piece_id=ATSAO7QMB43NHCL4IGW2APVOACK7FHU4LMQ4OIELBRNBWXEG3VHQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 188.166.225.170:54644->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016571960 (60h3m8.739135s) [noise=true , address=104.36.82.15:4273 , piece_id=A4M2WE67BHCHFYK7VYIICVVCHOMQBJAUMFQVP3U3AFOBC6EVGSSQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1cgTYjzrDTV5a4aohnpZvGPRweYkRv2dyKhQw46nQwMtAK3Kuv, address:104.36.82.15:4273): protocol: write tcp 188.166.225.170:57148->104.36.82.15:4273: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016605394 (60h3m8.739168434s) [noise=true , address=104.36.82.46:4481 , piece_id=2IBGLYHQR33S7VQ57EUVIM6TN26AWOPCOWYOVM2YU2YVRQHQZGBA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:57564->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016618650 (60h3m8.7391817s) [noise=true , address=77.90.31.21:4905 , piece_id=HHUOZXJUW6HHRB6PDZ6Z62K27DY27LB644UO4W2K5PI2HSJMH3NQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 188.166.225.170:44248->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016645120 (60h3m8.739208161s) [seg_pos={0 0}, upload=57602]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:38:59.246727247 (60h3m8.969290327s) [upload=57602]: Done uploading segments... Jul 26 23:38:59.246749859 (60h3m8.969312929s) [upload=57602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.232255691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.943785459s FAIL (2025-07-26 23:38:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.943810848s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X] duration: 30.001112162s FAIL (2025-07-26 23:38:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001134443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [.........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234209214s FAIL (2025-07-26 23:38:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== m24.391763459s) [noise=true , address=104.36.82.5:4840 , piece_id=R3KN7BXF5WUZBZV7JILT6BOXE32WNHZSO6HPIT5KVOIQOZKTWAVQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.669239934 (60h2m24.391802974s) [noise=true , address=104.36.82.14:4221 , piece_id=GXRK5RM4WE2KWQXLM25WNXSU7U7X5WJVFTGUTJ6XD5CPPUYT4UXQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.669507606 (60h2m24.392070646s) [noise=true , address=104.36.82.10:4078 , piece_id=CVRHX3UNI7DEMBXQ3YPTV23EMWEVWP5YM3QDDCDDVX3Q5YY57H6Q, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670240638 (60h2m24.392803678s) [noise=true , address=104.36.82.6:4875 , piece_id=7GEBDKM3ZNBO3NJBVQW3PI2VDRURMKM75UP5GJ3X6HZPIZJY3YMA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670374474 (60h2m24.392937514s) [noise=true , address=104.36.82.10:4077 , piece_id=57APRH5AKKCJFT2UL644YESH7CAVPXGR4BDUVRAZ2LTZ7FMOTGQA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670757387 (60h2m24.393320427s) [noise=true , address=77.90.31.16:4734 , piece_id=AZJ3LBGUQDAJCAQ7YMHJTZCMPADTTWZGNVA2OOJSTIACGHMBO2YA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.671048555 (60h2m24.393611595s) [noise=true , address=104.36.82.13:4187 , piece_id=M42HABRDVKHENZJVCR6P4XQIS4P2HZ23G34AZC2GDFESZLISKJ2A, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.671957163 (60h2m24.394520203s) [noise=true , address=104.36.82.3:4765 , piece_id=GYHURFAGQLPNAGKYIHDBOJ5JMEMFZULYS763DPDCGTH3KIPV3BIQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.672284149 (60h2m24.394847189s) [noise=true , address=104.36.82.13:4185 , piece_id=JIILB7TYWN625HZC3DSVQIWKNTXA6ZKSYBXIZV4FKZ75IFU4DT2Q, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.673948042 (60h2m24.396511082s) [noise=true , address=104.36.82.16:4293 , piece_id=OVSA675FVUNB3USAQ4DEZMUH6IQD6C5PQJQCYBPUXMFEQLZBAOVA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674232486 (60h2m24.396795526s) [noise=true , address=104.36.82.7:4912 , piece_id=GAG2M536ZN3NMECWQWDZI2WDXMUGE7Q2IMJ3XZORC7SSLV3FRBEQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674534263 (60h2m24.397097303s) [noise=true , address=104.36.82.8:4972 , piece_id=COM2PORJRNEIZBRMFMYJXKEHDXMXDBD7RLSVRFHYV4BXUBO3FJIQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674820932 (60h2m24.397383972s) [noise=true , address=45.91.116.10:4082 , piece_id=MF27QJDYHV22G44G43RD2T664Y43P3XM6BH26JEPKGYDMW4W3JCQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.675080538 (60h2m24.397643578s) [noise=true , address=45.91.116.8:4047 , piece_id=GHHFYWMOBEMJPAFVDI2ZRTQC2RVYFNYAP4CMPYEXRYTUXAIJVADQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:16.420628258 (60h2m26.143191299s) [upload=57606]: Next returned nil segment Jul 26 23:38:16.420645290 (60h2m26.14320832s) [upload=57606]: Waiting for error group managing segments... Jul 26 23:38:16.697463388 (60h2m26.420026438s) [noise=true , address=45.91.116.7:4009 , piece_id=J5NUILU5V7DOOYU6DZTIKGU64F64QITWMFLRN76LBUACWFJZRLMQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.708920047 (60h2m26.431483087s) [noise=true , address=77.90.31.16:4734 , piece_id=AZJ3LBGUQDAJCAQ7YMHJTZCMPADTTWZGNVA2OOJSTIACGHMBO2YA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.846707878 (60h2m26.569270918s) [noise=true , address=45.91.116.8:4047 , piece_id=GHHFYWMOBEMJPAFVDI2ZRTQC2RVYFNYAP4CMPYEXRYTUXAIJVADQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.871264612 (60h2m26.593827642s) [noise=true , address=45.91.116.10:4082 , piece_id=MF27QJDYHV22G44G43RD2T664Y43P3XM6BH26JEPKGYDMW4W3JCQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:43.795759508 (60h2m53.518322548s) [noise=true , address=104.36.82.10:4078 , piece_id=CVRHX3UNI7DEMBXQ3YPTV23EMWEVWP5YM3QDDCDDVX3Q5YY57H6Q, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:43776->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795812639 (60h2m53.518375679s) [noise=true , address=104.36.82.10:4079 , piece_id=7Z6D4HRRYMAQHE3JSBA4ATDUHYJ3RPC7OAZDSILSRAA7T3U3RUOQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:36432->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795846314 (60h2m53.518409354s) [noise=true , address=104.36.82.6:4875 , piece_id=7GEBDKM3ZNBO3NJBVQW3PI2VDRURMKM75UP5GJ3X6HZPIZJY3YMA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:59630->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795878636 (60h2m53.518441676s) [noise=true , address=104.36.82.3:4765 , piece_id=GYHURFAGQLPNAGKYIHDBOJ5JMEMFZULYS763DPDCGTH3KIPV3BIQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:42150->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795899916 (60h2m53.518462956s) [noise=true , address=104.36.82.14:4221 , piece_id=GXRK5RM4WE2KWQXLM25WNXSU7U7X5WJVFTGUTJ6XD5CPPUYT4UXQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:33240->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795977294 (60h2m53.518540324s) [noise=true , address=104.36.82.16:4293 , piece_id=OVSA675FVUNB3USAQ4DEZMUH6IQD6C5PQJQCYBPUXMFEQLZBAOVA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:58082->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796015548 (60h2m53.518578588s) [noise=true , address=104.36.82.8:4972 , piece_id=COM2PORJRNEIZBRMFMYJXKEHDXMXDBD7RLSVRFHYV4BXUBO3FJIQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:45332->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796051086 (60h2m53.518614126s) [noise=true , address=104.36.82.5:4840 , piece_id=R3KN7BXF5WUZBZV7JILT6BOXE32WNHZSO6HPIT5KVOIQOZKTWAVQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796087886 (60h2m53.518650926s) [noise=true , address=104.36.82.7:4912 , piece_id=GAG2M536ZN3NMECWQWDZI2WDXMUGE7Q2IMJ3XZORC7SSLV3FRBEQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:55634->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796125979 (60h2m53.518689019s) [noise=true , address=104.36.82.13:4187 , piece_id=M42HABRDVKHENZJVCR6P4XQIS4P2HZ23G34AZC2GDFESZLISKJ2A, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 188.166.225.170:34478->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796145877 (60h2m53.518708917s) [noise=true , address=104.36.82.10:4077 , piece_id=57APRH5AKKCJFT2UL644YESH7CAVPXGR4BDUVRAZ2LTZ7FMOTGQA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:45686->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796149734 (60h2m53.518712774s) [noise=true , address=104.36.82.13:4185 , piece_id=JIILB7TYWN625HZC3DSVQIWKNTXA6ZKSYBXIZV4FKZ75IFU4DT2Q, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:49934->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796189531 (60h2m53.518752571s) [seg_pos={0 0}, upload=57606]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:38:44.028591695 (60h2m53.751154735s) [upload=57606]: Done uploading segments... Jul 26 23:38:44.028617073 (60h2m53.751180113s) [upload=57606]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.234231737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X.] duration: 49.881669679s FAIL (2025-07-26 23:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.881690869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.230648197s FAIL (2025-07-26 23:38:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.230658908s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.838478015s FAIL (2025-07-26 23:38:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.838508092s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234574129s FAIL (2025-07-26 23:38:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.234600701s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233703339s FAIL (2025-07-26 23:37:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== :59.680062859 (60h1m9.402625899s) [noise=true , address=79.127.219.38:4015 , piece_id=JPSZ3NOQAAYJRKRGPWK3U7HUHD2SU7EFCNACKYE5K7ZE7WOS7DNA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680144155 (60h1m9.402707195s) [noise=true , address=79.127.205.235:4677 , piece_id=65OUDRPAZWTNUPPWOAKM6J73GZU7EL3K6UTGEDJ2M24XIGZ22FYA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680481611 (60h1m9.403044651s) [noise=true , address=77.90.31.18:4821 , piece_id=DJXMXCRUBMOB4SQACXOX6376MUA4CQRWEOJAK3N2E6DO6PFV3IPQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680610417 (60h1m9.403173457s) [noise=true , address=104.36.82.21:4493 , piece_id=Q52S6O65QUB3LSC66EPGY62ESVGB7RQZY2W7I6WUZK2Z4ILFDOQA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680784491 (60h1m9.403347531s) [noise=true , address=79.127.205.233:4639 , piece_id=MYHLASW5TLXLY6SNYA4GGP4WLMZJ6QEBPGZU3OVHO5GT2WWU4M4A, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680879031 (60h1m9.403442071s) [noise=true , address=104.36.82.47:4547 , piece_id=NFC2PLVBXUQZTEZZXRP7WMSIOPGJHWFDGHL35Q7NWQ74ZO2RBAVA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681046291 (60h1m9.403609331s) [noise=true , address=79.127.205.246:4274 , piece_id=YDQ4WQATQITXAGG4BXS6YSDR2BQRIZN3DYH2MFJJVTITQIYM54IQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681183113 (60h1m9.403746153s) [noise=true , address=77.90.31.16:4748 , piece_id=QMC2ZIYYHWQYGJFZGWGU4HE4CGT7DGCCWMJNF7JJKYWP4WL3AV6Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681305588 (60h1m9.403868628s) [noise=true , address=77.90.31.15:4720 , piece_id=2FUGVAEJINETPVOX2WODXHIMIPCSPLEVWDGT3W4AGSP4LTE3IZDQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681433672 (60h1m9.403996702s) [noise=true , address=104.36.82.46:4482 , piece_id=75EHPLZR3CHQSHLRSQKSBMWGKCNLZRPIL7R2TJTO3FW34RUY7KOQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681515028 (60h1m9.404078068s) [noise=true , address=45.91.116.5:4307 , piece_id=MDMUYA66DBRD25ERROMFIQSSKKDEG3LS4AYGJQI54LYHHWN64BGQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681655116 (60h1m9.404218156s) [noise=true , address=45.91.116.9:4277 , piece_id=YK7OIYGDM5YST3FNOSVFL3LBZGLDU7UU2LXP3URM74S4BNBWUEJA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681720541 (60h1m9.404283581s) [noise=true , address=77.90.31.12:4602 , piece_id=HMQPZFADUJ73QVP45RDVQ74ZWYLA7Q3ZHEKAITFCGOTDJSZSLZUQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681888082 (60h1m9.404451122s) [noise=true , address=104.36.82.54:4184 , piece_id=BXYUM7HMK4J5IIJJPVK5NXJH3TQE3JLGYAWCPYCITB6E3AM6HKSQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681951202 (60h1m9.404514242s) [noise=true , address=104.36.82.44:4872 , piece_id=ODYP6BQEMI7G25YWBM6D3JKI4DT5AADQFXONOIAH3SBJTKR7Z44Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682116008 (60h1m9.404679048s) [noise=true , address=64.49.13.138:4483 , piece_id=5P724JSFAKOINUQX733QCCKYUYSTFKVVRMO5HFCLFBJDIPI3JKHQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682191392 (60h1m9.404754422s) [noise=true , address=104.36.82.22:4538 , piece_id=UEHEDNUTQ6L2RWTZ25PFYOUIP4LILKWSMKOBBD5FC53TVCMGDHIA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682398979 (60h1m9.404962019s) [noise=true , address=77.90.31.17:4792 , piece_id=2ZA76STMDR63VA24ZZNPJKTE3CPLNRFIGWVAUITFLGRBSRBK7C2Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:37:00.789888640 (60h1m10.51245168s) [upload=57585]: Next returned nil segment Jul 26 23:37:00.789906635 (60h1m10.512469675s) [upload=57585]: Waiting for error group managing segments... Jul 26 23:37:00.956777980 (60h1m10.67934102s) [noise=true , address=64.49.13.138:4483 , piece_id=5P724JSFAKOINUQX733QCCKYUYSTFKVVRMO5HFCLFBJDIPI3JKHQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:37:01.070106992 (60h1m10.792670032s) [noise=true , address=79.127.205.233:4639 , piece_id=MYHLASW5TLXLY6SNYA4GGP4WLMZJ6QEBPGZU3OVHO5GT2WWU4M4A, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.099846674 (60h1m10.822409714s) [noise=true , address=79.127.205.246:4274 , piece_id=YDQ4WQATQITXAGG4BXS6YSDR2BQRIZN3DYH2MFJJVTITQIYM54IQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.135438291 (60h1m10.858001331s) [noise=true , address=79.127.205.235:4677 , piece_id=65OUDRPAZWTNUPPWOAKM6J73GZU7EL3K6UTGEDJ2M24XIGZ22FYA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.591445782 (60h1m11.314008822s) [noise=true , address=45.91.116.9:4277 , piece_id=YK7OIYGDM5YST3FNOSVFL3LBZGLDU7UU2LXP3URM74S4BNBWUEJA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.662548282 (60h1m11.385111322s) [noise=true , address=79.127.219.38:4015 , piece_id=JPSZ3NOQAAYJRKRGPWK3U7HUHD2SU7EFCNACKYE5K7ZE7WOS7DNA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.708791789 (60h1m11.431354829s) [noise=true , address=77.90.31.16:4748 , piece_id=QMC2ZIYYHWQYGJFZGWGU4HE4CGT7DGCCWMJNF7JJKYWP4WL3AV6Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.751670870 (60h1m11.47423391s) [noise=true , address=77.90.31.12:4602 , piece_id=HMQPZFADUJ73QVP45RDVQ74ZWYLA7Q3ZHEKAITFCGOTDJSZSLZUQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.772742276 (60h1m11.495305316s) [noise=true , address=77.90.31.18:4821 , piece_id=DJXMXCRUBMOB4SQACXOX6376MUA4CQRWEOJAK3N2E6DO6PFV3IPQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:07.721357463 (60h1m17.443920503s) [noise=true , address=45.91.116.5:4307 , piece_id=MDMUYA66DBRD25ERROMFIQSSKKDEG3LS4AYGJQI54LYHHWN64BGQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:58.546719995 (60h2m8.269283035s) [noise=true , address=77.90.31.15:4720 , piece_id=2FUGVAEJINETPVOX2WODXHIMIPCSPLEVWDGT3W4AGSP4LTE3IZDQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:37:58.546880763 (60h2m8.269443803s) [noise=true , address=104.36.82.46:4482 , piece_id=75EHPLZR3CHQSHLRSQKSBMWGKCNLZRPIL7R2TJTO3FW34RUY7KOQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547007456 (60h2m8.269570496s) [noise=true , address=104.36.82.47:4547 , piece_id=NFC2PLVBXUQZTEZZXRP7WMSIOPGJHWFDGHL35Q7NWQ74ZO2RBAVA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547041451 (60h2m8.269604491s) [noise=true , address=77.90.31.17:4792 , piece_id=2ZA76STMDR63VA24ZZNPJKTE3CPLNRFIGWVAUITFLGRBSRBK7C2Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: rpc: dial tcp 77.90.31.17:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547218600 (60h2m8.26978164s) [noise=true , address=104.36.82.22:4538 , piece_id=UEHEDNUTQ6L2RWTZ25PFYOUIP4LILKWSMKOBBD5FC53TVCMGDHIA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:104.36.82.22:4538): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:37:58.547269857 (60h2m8.269832897s) [noise=true , address=104.36.82.21:4493 , piece_id=Q52S6O65QUB3LSC66EPGY62ESVGB7RQZY2W7I6WUZK2Z4ILFDOQA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: rpc: dial tcp 104.36.82.21:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547316517 (60h2m8.269879557s) [noise=true , address=104.36.82.54:4184 , piece_id=BXYUM7HMK4J5IIJJPVK5NXJH3TQE3JLGYAWCPYCITB6E3AM6HKSQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG, address:104.36.82.54:4184): protocol: rpc: dial tcp 104.36.82.54:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547350221 (60h2m8.269913261s) [noise=true , address=104.36.82.44:4872 , piece_id=ODYP6BQEMI7G25YWBM6D3JKI4DT5AADQFXONOIAH3SBJTKR7Z44Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: write tcp 188.166.225.170:42584->104.36.82.44:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547363888 (60h2m8.269926928s) [seg_pos={0 0}, upload=57585]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:37:58.779170594 (60h2m8.501733634s) [upload=57585]: Done uploading segments... Jul 26 23:37:58.779189010 (60h2m8.50175205s) [upload=57585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.233718097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [..........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234316236s FAIL (2025-07-26 23:37:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 7579]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:37:15.228521305 (60h1m24.951084335s) [upload=57579]: Done uploading segments... Jul 26 23:37:15.228545440 (60h1m24.95110848s) [upload=57579]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:37:24.877659019 (60h1m34.600222059s) [upload=57593]: Starting upload Jul 26 23:37:24.877865344 (60h1m34.600428384s) [upload=57593]: Done starting upload Jul 26 23:37:24.877926822 (60h1m34.600489862s) [upload=57593]: Uploading segments... Jul 26 23:37:24.878036061 (60h1m34.600599101s) [upload=57593]: Got next segment. Inline: false Jul 26 23:37:24.878080616 (60h1m34.600643656s) [upload=57593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:37:24.878085335 (60h1m34.600648375s) [upload=57593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:37:25.110624903 (60h1m34.833187933s) [seg_pos={0 0}, upload=57593]: Begin upload segment... Jul 26 23:37:25.110663557 (60h1m34.833226597s) [seg_pos={0 0}, upload=57593]: Done begin upload segment. Jul 26 23:37:25.110688274 (60h1m34.833251314s) [noise=true , address=104.36.82.10:4079 , piece_id=CLZGOCKP6ZJFQPAJBIH4ACWPGRME46KDEWFONH5UHTNYGPHBL5TA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.110766153 (60h1m34.833329194s) [noise=true , address=104.36.82.4:4801 , piece_id=BZBEDAYL7IQL3WMQGV5WMX634AOVEPTBR33S5YND5F7O534PPDVQ, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.111029688 (60h1m34.833592728s) [noise=true , address=104.36.82.11:4115 , piece_id=UIFL6FI26LZRFQ3DFY723CFKCCHCZW3GOAUFVMBXCOKQ36DB5ETQ, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.111270268 (60h1m34.833833308s) [noise=true , address=77.90.31.15:4699 , piece_id=6DA3PZWN6PBL7NDCHL73G7TUJ3JU3NOGEFAUNYKHRMOSZUZZCOZA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.111508033 (60h1m34.834071063s) [noise=true , address=77.90.31.13:4628 , piece_id=XLAXOWLPJ5AFC3PUZLSBFPEVW67WQXBYG2YWDDJA3V2FFKHHPEXA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.111613415 (60h1m34.834176495s) [noise=true , address=45.91.116.8:4047 , piece_id=PCXCMSMVGJ5SONTP2YDA4FPA232U5BBWPREN3PZVKEQI4K7Q3J3A, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.111805532 (60h1m34.834368562s) [noise=true , address=77.90.31.21:4879 , piece_id=2K7G2C4EPDTB4Z7JD7JMDQXP5OPFR2KUEJXJK5WM2ZJVCMVGUZBA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.112063916 (60h1m34.834626956s) [noise=true , address=45.91.116.11:4119 , piece_id=3QGLRCNG5OURQGXCOAF5OJH7XZFX7NTREYR3W23NIHRXIUJID5EQ, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.112357819 (60h1m34.834920869s) [noise=true , address=45.91.116.7:4009 , piece_id=K5JQASAUTPSLFIAKO7WCE5Q27VHUS5OBYN4JJBTKT5VETWQENEUQ, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.112401792 (60h1m34.834964833s) [noise=true , address=77.90.31.14:4663 , piece_id=IVYJMW5FLNG5JUAKWWAFATKC2SQMUP7R6UQFK3MM7WKMOSF7ETYA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.112768174 (60h1m34.835331214s) [noise=true , address=104.36.82.2:4732 , piece_id=4BSEUB7WFUPRKHMJAHPYROYEUP27IEGLRWFDGE4K62236GOO6YOA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.112894475 (60h1m34.835457525s) [noise=true , address=45.91.116.14:4190 , piece_id=2U4QPEI42PCACDFPIQI7PIEZCXTNMKGBFY3QOW4KWHTCXEOHRO7Q, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.113024755 (60h1m34.835587795s) [noise=true , address=104.36.82.9:4007 , piece_id=2QIKK3DOZ7PCPNKRKOSNDIB6NFD52SWIPCWMVJ6MG4Q4MPLNWPZA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.113502198 (60h1m34.836065238s) [noise=true , address=104.36.82.15:4259 , piece_id=Q3CBMY5CHYCYQR5UW6KFWAI27U2A46SRMO67QOIKBXP52PHAB7NQ, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.113912122 (60h1m34.836475162s) [noise=true , address=45.91.116.13:4155 , piece_id=J7D7BP3I7GNCLBX4GCVMAHRGC7FABHONWNXSK2RYQGRG464FZDGA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:25.114216124 (60h1m34.836779164s) [noise=true , address=104.36.82.5:4837 , piece_id=7SCQJ2WRP7CYKPLXTHRM3U65NEY2ZUA3XKXUBMJGHORCQQUIODJA, seg_pos={0 0}, upload=57593]: Uploading piece... Jul 26 23:37:26.862404016 (60h1m36.584967056s) [upload=57593]: Next returned nil segment Jul 26 23:37:26.862741983 (60h1m36.585305023s) [upload=57593]: Waiting for error group managing segments... Jul 26 23:37:27.209758891 (60h1m36.932321931s) [noise=true , address=45.91.116.13:4155 , piece_id=J7D7BP3I7GNCLBX4GCVMAHRGC7FABHONWNXSK2RYQGRG464FZDGA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:27.210005863 (60h1m36.932568893s) [noise=true , address=45.91.116.11:4119 , piece_id=3QGLRCNG5OURQGXCOAF5OJH7XZFX7NTREYR3W23NIHRXIUJID5EQ, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:27.213284676 (60h1m36.935847716s) [noise=true , address=45.91.116.8:4047 , piece_id=PCXCMSMVGJ5SONTP2YDA4FPA232U5BBWPREN3PZVKEQI4K7Q3J3A, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:27.213760838 (60h1m36.936323878s) [noise=true , address=45.91.116.7:4009 , piece_id=K5JQASAUTPSLFIAKO7WCE5Q27VHUS5OBYN4JJBTKT5VETWQENEUQ, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:27.224095081 (60h1m36.946658121s) [noise=true , address=77.90.31.14:4663 , piece_id=IVYJMW5FLNG5JUAKWWAFATKC2SQMUP7R6UQFK3MM7WKMOSF7ETYA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:27.384391727 (60h1m37.106954767s) [noise=true , address=45.91.116.14:4190 , piece_id=2U4QPEI42PCACDFPIQI7PIEZCXTNMKGBFY3QOW4KWHTCXEOHRO7Q, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:28.570321528 (60h1m38.292884568s) [noise=true , address=77.90.31.13:4628 , piece_id=XLAXOWLPJ5AFC3PUZLSBFPEVW67WQXBYG2YWDDJA3V2FFKHHPEXA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:35.667085631 (60h1m45.389648671s) [noise=true , address=77.90.31.21:4879 , piece_id=2K7G2C4EPDTB4Z7JD7JMDQXP5OPFR2KUEJXJK5WM2ZJVCMVGUZBA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:53.022103981 (60h2m2.744667011s) [noise=true , address=77.90.31.15:4699 , piece_id=6DA3PZWN6PBL7NDCHL73G7TUJ3JU3NOGEFAUNYKHRMOSZUZZCOZA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: <nil> Jul 26 23:37:54.235808122 (60h2m3.958371192s) [noise=true , address=104.36.82.11:4115 , piece_id=UIFL6FI26LZRFQ3DFY723CFKCCHCZW3GOAUFVMBXCOKQ36DB5ETQ, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.235900358 (60h2m3.958463418s) [noise=true , address=104.36.82.15:4259 , piece_id=Q3CBMY5CHYCYQR5UW6KFWAI27U2A46SRMO67QOIKBXP52PHAB7NQ, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.235959852 (60h2m3.958522902s) [noise=true , address=104.36.82.5:4837 , piece_id=7SCQJ2WRP7CYKPLXTHRM3U65NEY2ZUA3XKXUBMJGHORCQQUIODJA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.236026900 (60h2m3.958589971s) [noise=true , address=104.36.82.10:4079 , piece_id=CLZGOCKP6ZJFQPAJBIH4ACWPGRME46KDEWFONH5UHTNYGPHBL5TA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.236098558 (60h2m3.958661628s) [noise=true , address=104.36.82.9:4007 , piece_id=2QIKK3DOZ7PCPNKRKOSNDIB6NFD52SWIPCWMVJ6MG4Q4MPLNWPZA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.236195904 (60h2m3.958758954s) [noise=true , address=104.36.82.4:4801 , piece_id=BZBEDAYL7IQL3WMQGV5WMX634AOVEPTBR33S5YND5F7O534PPDVQ, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.236265848 (60h2m3.958828938s) [noise=true , address=104.36.82.2:4732 , piece_id=4BSEUB7WFUPRKHMJAHPYROYEUP27IEGLRWFDGE4K62236GOO6YOA, seg_pos={0 0}, upload=57593]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:51272->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:54.236290635 (60h2m3.958853706s) [seg_pos={0 0}, upload=57593]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:37:54.468611666 (60h2m4.191174696s) [upload=57593]: Done uploading segments... Jul 26 23:37:54.468639810 (60h2m4.19120286s) [upload=57593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.23433926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X] duration: 1m0.000708213s FAIL (2025-07-26 23:37:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000725676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.835837934s FAIL (2025-07-26 23:37:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.83587271s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235133611s FAIL (2025-07-26 23:37:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235153118s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....................XXX.X..X.....XXXXX...XXX.X.....X.X..X..] duration: 7.358820394s FAIL (2025-07-26 23:37:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.35884342s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.23593907s FAIL (2025-07-26 23:37:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Jul 26 23:36:45.644114149 (60h0m55.366677179s) [upload=57579]: Starting upload Jul 26 23:36:45.644201647 (60h0m55.366764697s) [upload=57579]: Done starting upload Jul 26 23:36:45.644287891 (60h0m55.366850932s) [upload=57579]: Uploading segments... Jul 26 23:36:45.644383475 (60h0m55.366946515s) [upload=57579]: Got next segment. Inline: false Jul 26 23:36:45.644407199 (60h0m55.36697024s) [upload=57579]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:36:45.644411539 (60h0m55.366974579s) [upload=57579]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:36:46.097162828 (60h0m55.819725868s) [seg_pos={0 0}, upload=57579]: Begin upload segment... Jul 26 23:36:46.097203115 (60h0m55.819766155s) [seg_pos={0 0}, upload=57579]: Done begin upload segment. Jul 26 23:36:46.097225197 (60h0m55.819788237s) [noise=true , address=77.90.31.10:4520 , piece_id=4WG2TQVAPODXWEJXZHBWI37RLX6L37VG3UBXFWIQIVHRKADIZKRA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.097267368 (60h0m55.819830418s) [noise=true , address=45.91.116.14:4191 , piece_id=JTHSBJWFO2OLNFCURGMHVPF6WEQ546CTF4WWJHLCJBC36QFEPZSQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.097780720 (60h0m55.82034376s) [noise=true , address=45.91.116.12:4226 , piece_id=YPVPBHIO3YNHF246ZILKB6MPZOWUNUJWLXJFODRDUNFVVU53DHSA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.097943342 (60h0m55.820506442s) [noise=true , address=104.36.82.7:4912 , piece_id=O7RCUL5SVI632LVJME7HLFQ5W3IZ5RDRIFW65YFHWQRLKH724D7A, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.098214009 (60h0m55.820777039s) [noise=true , address=104.36.82.6:4876 , piece_id=ZECYWUVRGH64RIB4G34SU44UZXHJPOPNNDBYZ65TTDHAW3INDROA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.098456292 (60h0m55.821019322s) [noise=true , address=104.36.82.16:4293 , piece_id=QANFRJZCLWP53WWV5WLQYXFQHICMG3EWFJOZF6FDYQ5C5HC4HZIA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.098701952 (60h0m55.821264993s) [noise=true , address=45.91.116.10:4081 , piece_id=3ZKLKIGW3LP2E42FVXCG6AU4VTCA6RJVERCJL2PIO7SHYCS2MOCQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.098943204 (60h0m55.821506244s) [noise=true , address=104.36.82.12:4151 , piece_id=DGQLLLFJY3K2N4QXRDJR5X32TTW5444XQ2AVEJD62TSXRCBE7CEA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.099249650 (60h0m55.82181268s) [noise=true , address=45.91.116.5:4299 , piece_id=DGVW6D7CRB76HUMZP4MPOUFGDKWBBMFHSMX5EP2WDUQWPJEZYO7Q, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.099451427 (60h0m55.822014467s) [noise=true , address=45.91.116.4:4940 , piece_id=EE6C4K7A3Z7Q4HWBLSZOCGWSWD7Y72XT7YLOFIU4P25UUKEIARSQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.099637142 (60h0m55.822200182s) [noise=true , address=104.36.82.9:4006 , piece_id=ICO3QVLCJKINZSTQROF5LKXBDFWTB5YLIEE2WOFX4OADBM52MG5Q, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.099828418 (60h0m55.822391458s) [noise=true , address=45.91.116.11:4119 , piece_id=5AWFSYYCJ3X5EFMX3MBU7BRZQ34MZW2EJGZE3IMBZOOZFRF2N7KA, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.100098915 (60h0m55.822661946s) [noise=true , address=104.36.82.2:4731 , piece_id=FLQSAKGSPWIE4LBD2DA3INW4O5AFPTEYDJ6LVR6AH32Y4DBFQR2A, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.100346169 (60h0m55.822909209s) [noise=true , address=77.90.31.14:4661 , piece_id=5GDT7GW55YWF3ZRHGMMZIK7BGT3ZTKP44SRQQF4ZFE5ST4H7ZOAQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.100545691 (60h0m55.823108731s) [noise=true , address=104.36.82.11:4115 , piece_id=WG34W6OLJBR6UCFKIKHTYPOGFM5CWEK6GJDENCBKUP57FVNT35EQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:46.100743319 (60h0m55.823306349s) [noise=true , address=104.36.82.15:4258 , piece_id=LMWNA337FV6FFVPCCPHHU2JQ3A7WTN5RCCELS6EQQNS2TGXC4DWQ, seg_pos={0 0}, upload=57579]: Uploading piece... Jul 26 23:36:47.743017396 (60h0m57.465580446s) [upload=57579]: Next returned nil segment Jul 26 23:36:47.743040961 (60h0m57.465604011s) [upload=57579]: Waiting for error group managing segments... Jul 26 23:36:48.122029330 (60h0m57.84459237s) [noise=true , address=45.91.116.10:4081 , piece_id=3ZKLKIGW3LP2E42FVXCG6AU4VTCA6RJVERCJL2PIO7SHYCS2MOCQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.125495001 (60h0m57.848058041s) [noise=true , address=45.91.116.5:4299 , piece_id=DGVW6D7CRB76HUMZP4MPOUFGDKWBBMFHSMX5EP2WDUQWPJEZYO7Q, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.197451405 (60h0m57.920014445s) [noise=true , address=45.91.116.4:4940 , piece_id=EE6C4K7A3Z7Q4HWBLSZOCGWSWD7Y72XT7YLOFIU4P25UUKEIARSQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.210901679 (60h0m57.933464719s) [noise=true , address=45.91.116.14:4191 , piece_id=JTHSBJWFO2OLNFCURGMHVPF6WEQ546CTF4WWJHLCJBC36QFEPZSQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.253424187 (60h0m57.975987227s) [noise=true , address=77.90.31.14:4661 , piece_id=5GDT7GW55YWF3ZRHGMMZIK7BGT3ZTKP44SRQQF4ZFE5ST4H7ZOAQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.346431639 (60h0m58.068994679s) [noise=true , address=45.91.116.11:4119 , piece_id=5AWFSYYCJ3X5EFMX3MBU7BRZQ34MZW2EJGZE3IMBZOOZFRF2N7KA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:36:48.362680518 (60h0m58.085243558s) [noise=true , address=45.91.116.12:4226 , piece_id=YPVPBHIO3YNHF246ZILKB6MPZOWUNUJWLXJFODRDUNFVVU53DHSA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:37:11.202518565 (60h1m20.925081605s) [noise=true , address=77.90.31.10:4520 , piece_id=4WG2TQVAPODXWEJXZHBWI37RLX6L37VG3UBXFWIQIVHRKADIZKRA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: <nil> Jul 26 23:37:14.994516042 (60h1m24.717079072s) [noise=true , address=104.36.82.12:4151 , piece_id=DGQLLLFJY3K2N4QXRDJR5X32TTW5444XQ2AVEJD62TSXRCBE7CEA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994561729 (60h1m24.717124769s) [noise=true , address=104.36.82.11:4115 , piece_id=WG34W6OLJBR6UCFKIKHTYPOGFM5CWEK6GJDENCBKUP57FVNT35EQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:40216->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994585224 (60h1m24.717148264s) [noise=true , address=104.36.82.16:4293 , piece_id=QANFRJZCLWP53WWV5WLQYXFQHICMG3EWFJOZF6FDYQ5C5HC4HZIA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994629048 (60h1m24.717192078s) [noise=true , address=104.36.82.15:4258 , piece_id=LMWNA337FV6FFVPCCPHHU2JQ3A7WTN5RCCELS6EQQNS2TGXC4DWQ, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994658524 (60h1m24.717221564s) [noise=true , address=104.36.82.9:4006 , piece_id=ICO3QVLCJKINZSTQROF5LKXBDFWTB5YLIEE2WOFX4OADBM52MG5Q, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:54928->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994670797 (60h1m24.717233837s) [noise=true , address=104.36.82.6:4876 , piece_id=ZECYWUVRGH64RIB4G34SU44UZXHJPOPNNDBYZ65TTDHAW3INDROA, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994675597 (60h1m24.717238758s) [noise=true , address=104.36.82.7:4912 , piece_id=O7RCUL5SVI632LVJME7HLFQ5W3IZ5RDRIFW65YFHWQRLKH724D7A, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994710975 (60h1m24.717274005s) [noise=true , address=104.36.82.2:4731 , piece_id=FLQSAKGSPWIE4LBD2DA3INW4O5AFPTEYDJ6LVR6AH32Y4DBFQR2A, seg_pos={0 0}, upload=57579]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:56146->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:14.994729150 (60h1m24.71729219s) [seg_pos={0 0}, upload=57579]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:37:15.228521305 (60h1m24.951084335s) [upload=57579]: Done uploading segments... Jul 26 23:37:15.228545440 (60h1m24.95110848s) [upload=57579]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235961814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.850607341s FAIL (2025-07-26 23:37:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.850639302s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......................XXX.X..X.....XXXXX...XXX.X.....X.X..X.] duration: 24.468959277s FAIL (2025-07-26 23:37:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.468987741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.24098133s FAIL (2025-07-26 23:36:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Jul 26 23:35:59.176391451 (60h0m8.898954492s) [noise=true , address=104.36.82.12:4153 , piece_id=BJNWI5YJHS5BOPFOT2VWLGRF57L4GPKMYPATINNYVUMQWGDF32NA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.176722735 (60h0m8.899285775s) [noise=true , address=104.36.82.52:4097 , piece_id=SHARRAH4CPNLHDTKSTQW4JT56P5POUTNSJAMX43WPI6JUOWCARVA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.176766800 (60h0m8.89932983s) [noise=true , address=79.127.205.246:4284 , piece_id=A76QOIFGNC254UGCATWHZ4LKMLMNU6X4CKQWHDJUZZMQO3MKTPLA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.176969187 (60h0m8.899532227s) [noise=true , address=77.90.31.11:4558 , piece_id=FANDGWHH2QXLENUIJ5AB3YZIIXSI5ZDO5PJH522ASA2AYXPVODSQ, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.177382057 (60h0m8.899945087s) [noise=true , address=77.90.31.15:4717 , piece_id=EHQA4XBV72GK7Q6WGPAUSYVOI7A5MT6E4DTXWVPZMPMNREV5OXKA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.177728240 (60h0m8.90029127s) [noise=true , address=45.91.116.9:4277 , piece_id=PL4YVDWGOSKN4WWSGOWH652XF4NBP5FZULIUB6DKDJFDNQE37REQ, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.177749409 (60h0m8.900312449s) [noise=true , address=77.90.31.13:4651 , piece_id=ZDLJ7TXZL6A7UQL3WFUPDQYIXAYQHH6DVIFJMKN6ACYFF6CA3BDQ, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178015459 (60h0m8.900578499s) [noise=true , address=79.127.205.245:4252 , piece_id=25WLOI5Q43ALGXXX5NLDJL7SPJ2VI4SIF5NBDQPKQXC4ZBSHNATQ, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178082476 (60h0m8.900645516s) [noise=true , address=79.127.219.38:4015 , piece_id=5J6RGFEPYAVJI75OCIZD7O7MJNYHU36LMV3KYCVKAYG7DNUZ6NWA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178260047 (60h0m8.900823087s) [noise=true , address=104.36.82.56:4264 , piece_id=5HOTKL7MBMML3LZ3XLNAWXXOUXXXQ7CY2UBIIBC65NCMDC32Y5IA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178364366 (60h0m8.900927397s) [noise=true , address=104.36.82.13:4202 , piece_id=UUBIG7TTEI4CBQRXJAPQN5QDBNGGBEWT24KK2UZ5KP7UYKEEWZLA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178487672 (60h0m8.901050712s) [noise=true , address=79.127.205.235:4669 , piece_id=RXTZRKCOLRF5K4PGP37TZBASNA7QVA4JP4TZMWP6LCXFZVAW2Z7A, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178715858 (60h0m8.901278888s) [noise=true , address=104.36.82.47:4530 , piece_id=27ISVFTIG3Z3PVOZHX2NOTUOZEPAVT4OU5I3S46A24PQLZSGSTCA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178720909 (60h0m8.901283949s) [noise=true , address=45.91.116.14:4206 , piece_id=LSKWCGRDAABXVWQJCQUCVCKXG3JWVMG2TVATF2ZBFHBCCKP2ZWLA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.178949466 (60h0m8.901512496s) [noise=true , address=79.127.219.47:4061 , piece_id=XTCRSYFFVMV44KOYNBNJ4PNCEZ2OLJD5B6E3KDHZ27BAC5YNHYZA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.179070036 (60h0m8.901633076s) [noise=true , address=104.36.82.46:4483 , piece_id=WGMZ7VPAWFB5JDTIX5QPN5FFTHAYAXHZLASMEEOGOCBCH3BUNTDQ, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:35:59.179178954 (60h0m8.901741994s) [noise=true , address=104.36.82.42:4832 , piece_id=UZ4ZT4E2UBJZ7JU3GW4LAHXPOJVUBV3FN4GGWHEE27XZHM6IPXPA, seg_pos={0 0}, upload=57569]: Uploading piece... Jul 26 23:36:00.283831338 (60h0m10.006394378s) [upload=57569]: Next returned nil segment Jul 26 23:36:00.283845324 (60h0m10.006408364s) [upload=57569]: Waiting for error group managing segments... Jul 26 23:36:00.384119310 (60h0m10.10668239s) [noise=true , address=79.127.219.47:4061 , piece_id=XTCRSYFFVMV44KOYNBNJ4PNCEZ2OLJD5B6E3KDHZ27BAC5YNHYZA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 23:36:00.597131907 (60h0m10.319694947s) [noise=true , address=79.127.205.246:4284 , piece_id=A76QOIFGNC254UGCATWHZ4LKMLMNU6X4CKQWHDJUZZMQO3MKTPLA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:00.602259989 (60h0m10.324823019s) [noise=true , address=79.127.205.235:4669 , piece_id=RXTZRKCOLRF5K4PGP37TZBASNA7QVA4JP4TZMWP6LCXFZVAW2Z7A, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:00.651276321 (60h0m10.373839411s) [noise=true , address=79.127.205.245:4252 , piece_id=25WLOI5Q43ALGXXX5NLDJL7SPJ2VI4SIF5NBDQPKQXC4ZBSHNATQ, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:01.200816908 (60h0m10.923379948s) [noise=true , address=79.127.219.38:4015 , piece_id=5J6RGFEPYAVJI75OCIZD7O7MJNYHU36LMV3KYCVKAYG7DNUZ6NWA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:01.244013696 (60h0m10.966576736s) [noise=true , address=77.90.31.15:4717 , piece_id=EHQA4XBV72GK7Q6WGPAUSYVOI7A5MT6E4DTXWVPZMPMNREV5OXKA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:01.482449857 (60h0m11.205012897s) [noise=true , address=45.91.116.9:4277 , piece_id=PL4YVDWGOSKN4WWSGOWH652XF4NBP5FZULIUB6DKDJFDNQE37REQ, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:11.589705827 (60h0m21.312268867s) [noise=true , address=45.91.116.14:4206 , piece_id=LSKWCGRDAABXVWQJCQUCVCKXG3JWVMG2TVATF2ZBFHBCCKP2ZWLA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:27.708660707 (60h0m37.431223747s) [noise=true , address=79.127.201.215:4358 , piece_id=HIXVYPG5CXXHCRNSBYPBJCD6SXWJ2N45G6WBH5CGESQVAGY6PDLA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:49.560372525 (60h0m59.282935565s) [noise=true , address=77.90.31.13:4651 , piece_id=ZDLJ7TXZL6A7UQL3WFUPDQYIXAYQHH6DVIFJMKN6ACYFF6CA3BDQ, seg_pos={0 0}, upload=57569]: Done uploading piece. err: <nil> Jul 26 23:36:58.088096185 (60h1m7.810659225s) [noise=true , address=104.36.82.52:4097 , piece_id=SHARRAH4CPNLHDTKSTQW4JT56P5POUTNSJAMX43WPI6JUOWCARVA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: rpc: dial tcp 104.36.82.52:4097: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088145620 (60h1m7.81070865s) [noise=true , address=104.36.82.56:4264 , piece_id=5HOTKL7MBMML3LZ3XLNAWXXOUXXXQ7CY2UBIIBC65NCMDC32Y5IA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088214511 (60h1m7.810777601s) [noise=true , address=104.36.82.47:4530 , piece_id=27ISVFTIG3Z3PVOZHX2NOTUOZEPAVT4OU5I3S46A24PQLZSGSTCA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088267062 (60h1m7.810830102s) [noise=true , address=104.36.82.46:4483 , piece_id=WGMZ7VPAWFB5JDTIX5QPN5FFTHAYAXHZLASMEEOGOCBCH3BUNTDQ, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088287190 (60h1m7.810850231s) [noise=true , address=104.36.82.42:4832 , piece_id=UZ4ZT4E2UBJZ7JU3GW4LAHXPOJVUBV3FN4GGWHEE27XZHM6IPXPA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: rpc: dial tcp 104.36.82.42:4832: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088313030 (60h1m7.81087606s) [noise=true , address=77.90.31.11:4558 , piece_id=FANDGWHH2QXLENUIJ5AB3YZIIXSI5ZDO5PJH522ASA2AYXPVODSQ, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: read tcp 188.166.225.170:44908->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088349489 (60h1m7.810912529s) [noise=true , address=104.36.82.12:4153 , piece_id=BJNWI5YJHS5BOPFOT2VWLGRF57L4GPKMYPATINNYVUMQWGDF32NA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:104.36.82.12:4153): protocol: write tcp 188.166.225.170:49356->104.36.82.12:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088364589 (60h1m7.810927629s) [noise=true , address=104.36.82.13:4202 , piece_id=UUBIG7TTEI4CBQRXJAPQN5QDBNGGBEWT24KK2UZ5KP7UYKEEWZLA, seg_pos={0 0}, upload=57569]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: write tcp 188.166.225.170:59560->104.36.82.13:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:58.088388525 (60h1m7.810951555s) [seg_pos={0 0}, upload=57569]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:36:58.327878892 (60h1m8.050441932s) [upload=57569]: Done uploading segments... Jul 26 23:36:58.327903118 (60h1m8.050466158s) [upload=57569]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.240994255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......................XXX.X..X.....XXXXX...XXX.X.....X.X..X] duration: 30.000845086s FAIL (2025-07-26 23:36:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000867019s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....] duration: 27.684552279s FAIL (2025-07-26 23:36:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.684566988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.246646019s FAIL (2025-07-26 23:36:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== : false Jul 26 23:35:55.093403123 (60h0m4.815966163s) [upload=57566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:35:55.093409024 (60h0m4.815972064s) [upload=57566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:35:55.335348588 (60h0m5.057911628s) [seg_pos={0 0}, upload=57566]: Begin upload segment... Jul 26 23:35:55.335390518 (60h0m5.057953558s) [seg_pos={0 0}, upload=57566]: Done begin upload segment. Jul 26 23:35:55.335425224 (60h0m5.057988254s) [noise=true , address=45.91.116.10:4082 , piece_id=WKTWDISDULC6WV4G5RQDRY3EH5RFCRVXVUOOCSAIO2RTCZ2SJIVQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.335932254 (60h0m5.058495294s) [noise=true , address=45.91.116.7:4009 , piece_id=KTLRHY2VM3U3MJUJBFRZ757WYAET7C56EWT6SHWPEB4RYRPKN2LQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.336013309 (60h0m5.05857635s) [noise=true , address=45.91.116.8:4048 , piece_id=3N7P7O4M5ASVGYLLXABRVDKRUN5DM7LF6J4IDIAXGSECOYI322AQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.336575925 (60h0m5.059138966s) [noise=true , address=104.36.82.5:4838 , piece_id=JGDW3V6YRSXBJEPGPLTTT6LFSNU6YXIP2LR7FBND47KCLZPH7KIQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.336657892 (60h0m5.059220932s) [noise=true , address=104.36.82.6:4876 , piece_id=R6PES6IMU7HCMQCY3U2R4UQ672H4SF7MCBNVVJUNR37NYSNLZ7BQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.337053249 (60h0m5.059616289s) [noise=true , address=104.36.82.12:4150 , piece_id=NM3C5N7J5UV5OE65VQERBXBII6ZJ3AQY3YKDNTC3A2ZSDYMYDFLA, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.337150305 (60h0m5.059713345s) [noise=true , address=45.91.116.4:4937 , piece_id=CTS2NA4IPCB47QZ4ORYMTKYLAK4XMLMIHJ7WV33JSCSLHPAL5JWA, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.337903807 (60h0m5.060466847s) [noise=true , address=45.91.116.6:4973 , piece_id=WFCXETEVRY53UARB4VAWTHGGDECMNXEETJOKUM5JJ3GHMK276RMQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.338136111 (60h0m5.060699151s) [noise=true , address=104.36.82.3:4767 , piece_id=2D2XWBAQT3NRUY7BR3BQAHWCMM23ZCNV2GTTSGS6DYZYIS54BA2A, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.338439571 (60h0m5.061002611s) [noise=true , address=104.36.82.15:4257 , piece_id=77L64RQTGG5I2KYPY5G5LK7UZYIH5PKAHR52VMNG7PD5WTM5MECA, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.338512381 (60h0m5.061075421s) [noise=true , address=104.36.82.9:4007 , piece_id=U2UR2LNJQZ5F5XAMHWJULU4XUJGCH7I3AHJW2GUUSODMD66OF5CQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.338547869 (60h0m5.061110909s) [noise=true , address=104.36.82.7:4912 , piece_id=HGJSW5AEDDE52F25CFKFUHKBT7K6SG272ZTBQGTIFHVCKJNQT3NQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.338923387 (60h0m5.061486428s) [noise=true , address=104.36.82.16:4295 , piece_id=522IC4QIA5BP2PXELL2LIT3PD435XZUMM2UFKY3IGQJQSKFVCU4A, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.339018709 (60h0m5.061581749s) [noise=true , address=45.91.116.5:4297 , piece_id=MBXZR5J6GTTX664575CZFXM7VHZDUZ2CINSICXRNXDNWF3ZJ54AA, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.339519688 (60h0m5.062082718s) [noise=true , address=104.36.82.6:4873 , piece_id=7WAEJ6UOFCEBO4AKGPYCU3OHIQWGGHYRQ7BNFR5QAGENRFAO6MZQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:55.339721004 (60h0m5.062284034s) [noise=true , address=104.36.82.11:4114 , piece_id=J6FG74AFCOLAH3TISWG45PWWUD4CJNWWKN33HQZSZUUBA7LXPAQQ, seg_pos={0 0}, upload=57566]: Uploading piece... Jul 26 23:35:57.074377644 (60h0m6.796940684s) [upload=57566]: Next returned nil segment Jul 26 23:35:57.074396490 (60h0m6.79695953s) [upload=57566]: Waiting for error group managing segments... Jul 26 23:35:57.360159811 (60h0m7.082722851s) [noise=true , address=45.91.116.10:4082 , piece_id=WKTWDISDULC6WV4G5RQDRY3EH5RFCRVXVUOOCSAIO2RTCZ2SJIVQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:35:57.427748730 (60h0m7.15031177s) [noise=true , address=45.91.116.8:4048 , piece_id=3N7P7O4M5ASVGYLLXABRVDKRUN5DM7LF6J4IDIAXGSECOYI322AQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:35:57.430007201 (60h0m7.152570241s) [noise=true , address=45.91.116.4:4937 , piece_id=CTS2NA4IPCB47QZ4ORYMTKYLAK4XMLMIHJ7WV33JSCSLHPAL5JWA, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:35:57.442541954 (60h0m7.165104994s) [noise=true , address=45.91.116.6:4973 , piece_id=WFCXETEVRY53UARB4VAWTHGGDECMNXEETJOKUM5JJ3GHMK276RMQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:35:57.456657180 (60h0m7.17922022s) [noise=true , address=45.91.116.7:4009 , piece_id=KTLRHY2VM3U3MJUJBFRZ757WYAET7C56EWT6SHWPEB4RYRPKN2LQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:35:57.842798996 (60h0m7.565362026s) [noise=true , address=45.91.116.5:4297 , piece_id=MBXZR5J6GTTX664575CZFXM7VHZDUZ2CINSICXRNXDNWF3ZJ54AA, seg_pos={0 0}, upload=57566]: Done uploading piece. err: <nil> Jul 26 23:36:24.445026770 (60h0m34.16758981s) [noise=true , address=104.36.82.6:4876 , piece_id=R6PES6IMU7HCMQCY3U2R4UQ672H4SF7MCBNVVJUNR37NYSNLZ7BQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445097556 (60h0m34.167660606s) [noise=true , address=104.36.82.12:4150 , piece_id=NM3C5N7J5UV5OE65VQERBXBII6ZJ3AQY3YKDNTC3A2ZSDYMYDFLA, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445154726 (60h0m34.167717776s) [noise=true , address=104.36.82.5:4838 , piece_id=JGDW3V6YRSXBJEPGPLTTT6LFSNU6YXIP2LR7FBND47KCLZPH7KIQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445212406 (60h0m34.167775456s) [noise=true , address=104.36.82.11:4114 , piece_id=J6FG74AFCOLAH3TISWG45PWWUD4CJNWWKN33HQZSZUUBA7LXPAQQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445223908 (60h0m34.167786959s) [noise=true , address=104.36.82.7:4912 , piece_id=HGJSW5AEDDE52F25CFKFUHKBT7K6SG272ZTBQGTIFHVCKJNQT3NQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445250498 (60h0m34.167813538s) [noise=true , address=104.36.82.16:4295 , piece_id=522IC4QIA5BP2PXELL2LIT3PD435XZUMM2UFKY3IGQJQSKFVCU4A, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445269836 (60h0m34.167832876s) [noise=true , address=104.36.82.15:4257 , piece_id=77L64RQTGG5I2KYPY5G5LK7UZYIH5PKAHR52VMNG7PD5WTM5MECA, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445302187 (60h0m34.167865227s) [noise=true , address=104.36.82.9:4007 , piece_id=U2UR2LNJQZ5F5XAMHWJULU4XUJGCH7I3AHJW2GUUSODMD66OF5CQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445338628 (60h0m34.167901658s) [noise=true , address=104.36.82.6:4873 , piece_id=7WAEJ6UOFCEBO4AKGPYCU3OHIQWGGHYRQ7BNFR5QAGENRFAO6MZQ, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 188.166.225.170:55074->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445347454 (60h0m34.167910494s) [noise=true , address=104.36.82.3:4767 , piece_id=2D2XWBAQT3NRUY7BR3BQAHWCMM23ZCNV2GTTSGS6DYZYIS54BA2A, seg_pos={0 0}, upload=57566]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:24.445363946 (60h0m34.167926986s) [seg_pos={0 0}, upload=57566]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:36:24.689242161 (60h0m34.411805201s) [upload=57566]: Done uploading segments... Jul 26 23:36:24.689268431 (60h0m34.411831471s) [upload=57566]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.246658182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236017235s FAIL (2025-07-26 23:36:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.236028698s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.944530294s FAIL (2025-07-26 23:36:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.94455432s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X....] duration: 51.44731203s FAIL (2025-07-26 23:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.447357839s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)