Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X..XXXXXXXXXX.XX.XXXXXX......X.......XX................] duration: 5.938822289s FAIL (2025-08-16 16:13:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.93884946s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.086746365s FAIL (2025-08-16 16:13:12 UTC) orbiter warning: context deadline exceeded Duration: 6.086764208s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X......................X.................X..............] duration: 8.362031113s FAIL (2025-08-16 16:13:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 8.362057564s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 6.146495219s FAIL (2025-08-16 16:12:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.146522771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X..XXXXXXXXXX.XX.XXXXXX......X.......XX...............] duration: 5.929392s FAIL (2025-08-16 16:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.92941732s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.875360087s FAIL (2025-08-16 16:12:30 UTC) orbiter warning: context deadline exceeded Duration: 5.875377441s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X......................X.................X.............] duration: 12.368399092s FAIL (2025-08-16 16:12:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 12.368427887s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X........] duration: 10.89348987s FAIL (2025-08-16 16:12:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 10.893511341s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX..............] duration: 6.156936595s FAIL (2025-08-16 16:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.156959078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.563440166s FAIL (2025-08-16 16:12:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.563461346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 6.024453385s FAIL (2025-08-16 16:11:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.024485285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X.....X......................X.................X............] duration: 13.510804066s FAIL (2025-08-16 16:11:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 13.510832682s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.363619712s FAIL (2025-08-16 16:11:22 UTC) orbiter warning: context deadline exceeded Duration: 6.363638257s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.......] duration: 16.808343968s FAIL (2025-08-16 16:11:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 16.808369536s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX.............] duration: 5.949188874s FAIL (2025-08-16 16:11:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.949211669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 6.02144588s FAIL (2025-08-16 16:11:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.021469476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.411097644s FAIL (2025-08-16 16:11:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.411119527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [.X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X......] duration: 16.94308239s FAIL (2025-08-16 16:10:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 16.943100515s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.359424326s FAIL (2025-08-16 16:10:27 UTC) orbiter warning: context deadline exceeded Duration: 6.359444926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X.....X......................X.................X...........] duration: 16.361090806s FAIL (2025-08-16 16:10:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 16.361116164s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX............] duration: 5.770557767s FAIL (2025-08-16 16:10:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.770606831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 6.022418939s FAIL (2025-08-16 16:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.022447967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X......................X.................X..........] duration: 8.727726545s FAIL (2025-08-16 16:09:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 8.727753286s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.613557951s FAIL (2025-08-16 16:09:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.613579704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX...........] duration: 6.028594379s FAIL (2025-08-16 16:09:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.028620939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 6.017694712s FAIL (2025-08-16 16:09:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.017720371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.345810556s FAIL (2025-08-16 16:09:24 UTC) orbiter warning: context deadline exceeded Duration: 6.345827939s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.....] duration: 9.347412643s FAIL (2025-08-16 16:09:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 9.347439014s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [X..X.....X......................X.................X.........] duration: 15.624695866s FAIL (2025-08-16 16:09:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 15.624721444s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 5.898204683s FAIL (2025-08-16 16:08:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.89822871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.639855971s FAIL (2025-08-16 16:08:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.639872071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX..........] duration: 5.874195033s FAIL (2025-08-16 16:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.8742194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X....] duration: 16.423876743s FAIL (2025-08-16 16:08:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 16.423894466s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration: 5.995035135s FAIL (2025-08-16 16:08:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.995056965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.37183842s FAIL (2025-08-16 16:08:13 UTC) orbiter warning: context deadline exceeded Duration: 6.371855732s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X...] duration: 10.032032429s FAIL (2025-08-16 16:08:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 10.032057147s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.X..X.....X......................X.................X........] duration: 7.864118267s FAIL (2025-08-16 16:07:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 7.864143795s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.511585375s FAIL (2025-08-16 16:07:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.511670268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [..X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X..] duration: 16.691376025s FAIL (2025-08-16 16:07:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 16.691403609s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX.........] duration: 5.910056728s FAIL (2025-08-16 16:07:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.910082498s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.357741772s FAIL (2025-08-16 16:07:32 UTC) orbiter warning: context deadline exceeded Duration: 6.35776115s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.443924315s FAIL (2025-08-16 16:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.444511981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration: 5.975438772s FAIL (2025-08-16 16:07:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.975467807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [X.X..X.....X......................X.................X.......] duration: 17.794706103s FAIL (2025-08-16 16:07:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 17.794735057s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.X.X..X.....X......................X.................X......] duration: 21.495178624s FAIL (2025-08-16 16:06:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 21.495203903s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [...X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.] duration: 12.426407278s FAIL (2025-08-16 16:06:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 12.426432226s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.481642927s FAIL (2025-08-16 16:06:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.481666925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 5.981195438s FAIL (2025-08-16 16:06:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.981224454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [......X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX........] duration: 8.051873517s FAIL (2025-08-16 16:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 8.051894817s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.016563107s FAIL (2025-08-16 16:06:18 UTC) orbiter warning: context deadline exceeded Duration: 6.016584227s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)