Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.668448426s FAIL (2025-06-26 06:33:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.668474517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......................X........................] duration: 5.948473317s FAIL (2025-06-26 06:33:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.948803625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 5.649145711s FAIL (2025-06-26 06:33:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.649172043s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.881295261s FAIL (2025-06-26 06:32:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.881320459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 5.719830591s FAIL (2025-06-26 06:32:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.719853135s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......................X.......................] duration: 6.134803055s FAIL (2025-06-26 06:32:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.134834176s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......................X......................] duration: 5.939452607s FAIL (2025-06-26 06:32:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.939476534s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.822937335s FAIL (2025-06-26 06:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.822960801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 5.628113927s FAIL (2025-06-26 06:31:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.628139375s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..............X......................X.....................] duration: 5.919087864s FAIL (2025-06-26 06:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.919145927s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.780549347s FAIL (2025-06-26 06:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.780572552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX..............X......................X....................] duration: 5.95393268s FAIL (2025-06-26 06:30:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.953952227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 5.70185789s FAIL (2025-06-26 06:30:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.701892156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX..............X......................X...................] duration: 5.9327479s FAIL (2025-06-26 06:30:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.932791806s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.877210799s FAIL (2025-06-26 06:30:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.877233913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 5.663657107s FAIL (2025-06-26 06:30:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.663685783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX..............X......................X..................] duration: 5.984294395s FAIL (2025-06-26 06:29:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.984318592s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 5.679031099s FAIL (2025-06-26 06:29:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.679052029s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.695649898s FAIL (2025-06-26 06:29:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.695671179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.69007493s FAIL (2025-06-26 06:29:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.6901059s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX..............X......................X.................] duration: 6.253336204s FAIL (2025-06-26 06:28:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.253357033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX..............X......................X................] duration: 6.5426951s FAIL (2025-06-26 06:28:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.542719358s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.08065376s FAIL (2025-06-26 06:28:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.080665033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 5.678560269s FAIL (2025-06-26 06:27:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.678589997s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX..............X......................X...............] duration: 6.033263806s FAIL (2025-06-26 06:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.033283623s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.822634688s FAIL (2025-06-26 06:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.822654345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX..............X......................X..............] duration: 5.899896544s FAIL (2025-06-26 06:27:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.899912475s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.733423955s FAIL (2025-06-26 06:27:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.733452931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.816213081s FAIL (2025-06-26 06:27:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.816233681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.782126572s FAIL (2025-06-26 06:26:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.782164485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX..............X......................X.............] duration: 5.886441205s FAIL (2025-06-26 06:26:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.886465591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX..............X......................X............] duration: 5.853511284s FAIL (2025-06-26 06:26:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.853533247s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.835234189s FAIL (2025-06-26 06:26:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.835255722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.566224561s FAIL (2025-06-26 06:26:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.566251003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 5.664712432s FAIL (2025-06-26 06:25:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.664741148s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX..............X......................X...........] duration: 6.079182791s FAIL (2025-06-26 06:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.079207398s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.090163405s FAIL (2025-06-26 06:25:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090181911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.544254159s FAIL (2025-06-26 06:25:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.544283315s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX..............X......................X..........] duration: 5.935172566s FAIL (2025-06-26 06:25:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93518482s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX..............X......................X.........] duration: 5.986745331s FAIL (2025-06-26 06:24:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.986781512s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.564050303s FAIL (2025-06-26 06:24:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.564077694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.027250332s FAIL (2025-06-26 06:24:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.027273486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX..............X......................X........] duration: 5.960315862s FAIL (2025-06-26 06:24:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.960336032s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.707869885s FAIL (2025-06-26 06:23:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.707892288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 5.620468309s FAIL (2025-06-26 06:23:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.620490683s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX..............X......................X.......] duration: 5.875213081s FAIL (2025-06-26 06:23:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.875235584s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.724994869s FAIL (2025-06-26 06:22:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.725016021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.816809005s FAIL (2025-06-26 06:22:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.816825857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.925923814s FAIL (2025-06-26 06:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.925966717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX..............X......................X......] duration: 5.893175387s FAIL (2025-06-26 06:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.893196689s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)