Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.705380971s
FAIL (2025-10-02 06:53:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.705414003s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.559958382s
FAIL (2025-10-02 06:53:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.559993158s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.581967334s
FAIL (2025-10-02 06:52:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.581993864s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.341615331s
FAIL (2025-10-02 06:52:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.341716214s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.781160855s
FAIL (2025-10-02 06:52:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.781322893s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.174986818s
FAIL (2025-10-02 06:52:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.175016563s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.163671909s
FAIL (2025-10-02 06:51:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.163703951s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.8515043s
FAIL (2025-10-02 06:51:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.851533465s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.358263373s
FAIL (2025-10-02 06:51:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.358292007s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.108592057s
FAIL (2025-10-02 06:50:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.10862001s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.882066506s
FAIL (2025-10-02 06:50:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.882081825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.244235488s
FAIL (2025-10-02 06:50:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.244303136s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.608162514s
FAIL (2025-10-02 06:50:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.608291409s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.406078301s
FAIL (2025-10-02 06:49:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.406119548s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.93961607s
FAIL (2025-10-02 06:49:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.939693667s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.452068244s
FAIL (2025-10-02 06:49:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.452104543s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
5.587377259s
FAIL (2025-10-02 06:49:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.587399552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.568363929s
FAIL (2025-10-02 06:48:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.56841705s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.191680082s
FAIL (2025-10-02 06:48:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.191712413s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.804861005s
FAIL (2025-10-02 06:48:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.804881495s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.276748884s
FAIL (2025-10-02 06:48:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.2767729s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.779580324s
FAIL (2025-10-02 06:48:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.779616043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.204642676s
FAIL (2025-10-02 06:47:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.204676661s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.781884262s
FAIL (2025-10-02 06:47:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.781911675s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.115370947s
FAIL (2025-10-02 06:47:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.115399371s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.550747826s
FAIL (2025-10-02 06:47:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.550778675s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.654600635s
FAIL (2025-10-02 06:46:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.654630231s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.224779339s
FAIL (2025-10-02 06:46:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.224808294s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
7.791793698s
FAIL (2025-10-02 06:46:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 7.791853231s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.663512593s
FAIL (2025-10-02 06:46:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.663545465s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.433971342s
FAIL (2025-10-02 06:46:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.434001289s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
50.64284033s
FAIL (2025-10-02 06:45:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.642897099s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.881726672s
FAIL (2025-10-02 06:45:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.881754084s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.255438779s
FAIL (2025-10-02 06:45:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
2 06:45:01.165728558 (209h13m46.627821051s) [upload=183988]: Got next segment. Inline: false
Oct 2 06:45:01.165773403 (209h13m46.627865886s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 2 06:45:01.165780918 (209h13m46.627873401s) [upload=183988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 2 06:45:01.431617535 (209h13m46.893710028s) [seg_pos={0 0}, upload=183988]: Begin upload segment...
Oct 2 06:45:01.431697767 (209h13m46.89379025s) [seg_pos={0 0}, upload=183988]: Done begin upload segment.
Oct 2 06:45:01.431733565 (209h13m46.893826048s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.431781507 (209h13m46.89387399s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.432603612 (209h13m46.894696095s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433107522 (209h13m46.895199995s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.433549994 (209h13m46.895642477s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.434770508 (209h13m46.896862991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435293313 (209h13m46.897385796s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.435758981 (209h13m46.897851464s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.436566698 (209h13m46.898659182s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437037004 (209h13m46.899129487s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437501258 (209h13m46.899593741s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.437998455 (209h13m46.900090938s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439162861 (209h13m46.901255345s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.439750080 (209h13m46.901842563s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.441380735 (209h13m46.903473208s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:01.443563041 (209h13m46.905655524s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Uploading piece...
Oct 2 06:45:03.164025603 (209h13m48.626118096s) [upload=183988]: Next returned nil segment
Oct 2 06:45:03.164074216 (209h13m48.626166699s) [upload=183988]: Waiting for error group managing segments...
Oct 2 06:45:03.564517516 (209h13m49.026609999s) [noise=true , address=45.91.116.6:4975 , piece_id=ZNB45P2Q3Q3LUS3DG4FPV5IHHBGSCE5VBCXCW3HFL56GN6EKO7JQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.610247563 (209h13m49.072340046s) [noise=true , address=77.90.31.21:4878 , piece_id=6S3KSKZN7QKWQCSAOJHVNCLET7JBWPZIRNFQI2SX3UKDMT732KLA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.616241056 (209h13m49.078333549s) [noise=true , address=45.91.116.11:4118 , piece_id=KBHSEEDCAKKH76W7FBETEES4UXAMTTA2AQAHETDNJ6ZND73X2OGQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.632371971 (209h13m49.094464475s) [noise=true , address=77.90.31.12:4589 , piece_id=OVKIKUBOCPLT3VJVV4ML6QIQLGOA2S4AIBUUR5IYKAUE2IQVUDGA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.639558567 (209h13m49.10165105s) [noise=true , address=45.91.116.12:4227 , piece_id=MGYI2TR4MPA6DKGFYIVMGTGLD5DO5BROGFCKWOP4Z4RNRWOSBPMQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.649768408 (209h13m49.111860891s) [noise=true , address=77.90.31.11:4554 , piece_id=J5OKH5FR5YIBUZEN55CPSKDJ7L3O7PGF5VDS2X3C7CGTBCXUEVUA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.654368487 (209h13m49.11646099s) [noise=true , address=45.91.116.5:4299 , piece_id=2YUDNHDTW3DDWZJPHUV4AHHAO6GJI3TVUKMXO3JDBDIG3YB5XLEA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.688906413 (209h13m49.150998906s) [noise=true , address=45.91.116.4:4939 , piece_id=MGTGITWIRGG53CWAY7276LOHSVJGACIAAMSISMKPZGJLECG52PLQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.706565178 (209h13m49.168657671s) [noise=true , address=45.91.116.8:4047 , piece_id=ZSWJVTA7HW7U2IHRKGKDUHNYCCSMK3YJ3KJBQLOIQ773XCGLWV3Q, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.744669075 (209h13m49.206761558s) [noise=true , address=45.91.116.13:4155 , piece_id=JMRFSKD77UQKZXUZIFCQPAAVCNFDNGNV7U2SKSAJDOKARS6NKA5A, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:03.767094667 (209h13m49.22918715s) [noise=true , address=45.91.116.8:4048 , piece_id=FVQDCO7EYOYOAOORYY7DAQYIAXWBBDN7OF7E5B2Z3MT34QUGGLUQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: <nil>
Oct 2 06:45:30.458020428 (209h14m15.920112912s) [noise=true , address=145.79.227.13:4801 , piece_id=ZL7KE3QVM63VCBNMAUEXELEERW2JC65RTCVUUQKYLFTEJZBZPRVA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42996->145.79.227.13:4801: use of closed network connection
Oct 2 06:45:30.458236099 (209h14m15.920328572s) [noise=true , address=145.79.227.5:4221 , piece_id=SWGYPEIGFBLMMVAUFNIEQ63SL5SEDXFOCGIFCB7C7ZNCDYDZS7BQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42060->145.79.227.5:4221: use of closed network connection
Oct 2 06:45:30.458311102 (209h14m15.920403585s) [noise=true , address=145.79.227.2:4115 , piece_id=DZADXAOVVZSNVIMHVE5N7XA4U7Y2RS6U6JQLTC2X6YSWWSGVP6YQ, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55776->145.79.227.2:4115: use of closed network connection
Oct 2 06:45:30.458404109 (209h14m15.920496592s) [noise=true , address=145.79.227.11:4730 , piece_id=QEESDYHFF4TTJ6DOW5BAXORGOSWAKLVFDUWMYZT5POSJ6J5D7IOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->145.79.227.11:4730: use of closed network connection
Oct 2 06:45:30.458519669 (209h14m15.920612142s) [noise=true , address=145.79.227.16:4909 , piece_id=DVOXCRCAGC43BIDHGJOGQBZLDM4QBUONXGWF3GVVUZ4V5YSC6NOA, seg_pos={0 0}, upload=183988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.16:4909: use of closed network connection
Oct 2 06:45:30.458566669 (209h14m15.920659152s) [seg_pos={0 0}, upload=183988]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 2 06:45:30.710989853 (209h14m16.173082346s) [upload=183988]: Done uploading segments...
Oct 2 06:45:30.711022285 (209h14m16.173114778s) [upload=183988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.255463786s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001535781s
FAIL (2025-10-02 06:45:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001569076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.231403461s
FAIL (2025-10-02 06:45:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.231430712s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.381707926s
FAIL (2025-10-02 06:44:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.381738855s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.354995452s
FAIL (2025-10-02 06:43:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.35501599s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.144639452s
FAIL (2025-10-02 06:43:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.144669269s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.451525127s
FAIL (2025-10-02 06:42:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.451547921s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.341714409s
FAIL (2025-10-02 06:42:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.34174636s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
13.371832149s
FAIL (2025-10-02 06:41:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.371886843s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000523944s
FAIL (2025-10-02 06:40:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00054823s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.769573823s
FAIL (2025-10-02 05:50:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.769605173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.74072823s
FAIL (2025-10-02 05:49:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.740756384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.691612676s
FAIL (2025-10-02 05:48:33 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.691626031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.782872357s
FAIL (2025-10-02 05:47:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.782899949s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.746015059s
FAIL (2025-10-02 05:47:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.746270404s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.68466484s
FAIL (2025-10-02 05:46:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.684693203s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.498393403s
FAIL (2025-10-02 05:46:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.498428689s
core
uplink
us-select-1
us1-placement-20
|